Source-linked AI summary
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
William L. Hamilton, Jure Leskovec, Dan Jurafsky
TL;DR
Historical evidence about changing word meanings is scarce, and the roles of frequency and polysemy remain unresolved. The paper evaluates diachronic embedding methods and applies them across multilingual historical corpora, finding that frequent words change more slowly while polysemous words change faster. These findings support statistical laws of semantic change and motivate distributional methods for historical research.
Problem
The roles of word frequency and polysemy in semantic change remain unanswered, while existing diachronic evidence relies mainly on case studies of a few words.
Method
The study compares PPMI, SVD, and word2vec embeddings against novel benchmarks and models frequency and polysemy effects while controlling for temporal trends and correlated repeated measurements.
Results
The analysis proposes two laws: frequent words change more slowly, while polysemous words change faster and may contribute to semantic change.
Takeaways & Limitations
Distributional methods can reveal statistical laws of semantic change and provide a methodology for future historical research.
Takeaways & Limitations
The findings do not establish causal links, and the contextual-diversity proxy for polysemy is slightly biased toward discourse function words.
Abstract
from arXiv · showhide
Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test. Word embeddings show promise as a diachronic tool, but have not been carefully evaluated. We develop a robust methodology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evolution. Using six historical corpora spanning four languages and two centuries, we propose two quantitative laws of semantic change: (i) the law of conformity---the rate of semantic change scales with an inverse power-law of word frequency; (ii) the law of innovation---independent of frequency, words that are more polysemous have higher rates of semantic change.
1 Introduction
Semantic change shows systematic variation, but the roles of frequency and polysemy remain unresolved because historical evidence is limited. The paper addresses this gap by evaluating distributional embeddings across broader, cross-linguistic data.
- Historical semantic change varies systematically across words, from stable meanings such as cat to the diverse meanings of cast.
- Frequency’s role in meaning change remains unresolved because it is associated with both faster and slower forms of linguistic change.
- Polysemous words may be more or less likely to change, but this relationship must be assessed while controlling for frequency.
- Distributional semantics represents words as vectors from co-occurrence relationships and compares their embeddings across time-periods.
- The study evaluates PPMI, SVD, and word2vec embeddings against novel benchmarks, then analyzes 6 corpora spanning 200 years and 4 languages.
- The analysis proposes two laws: semantic change decreases with word frequency, while polysemous words change faster independently of frequency.
2 Diachronic embedding methods
Diachronic embeddings are built separately for each historical period, aligned across time, and analyzed through pairwise similarity shifts or individual-word displacement. The methods include PPMI, SVD, and SGNS, applied to six historical datasets with explicit corpus and alignment choices.
- 2.1 Embedding methods: Models represent words through co-occurrence statistics using PPMI, SVD, or SGNS, with semantic similarity approximated by cosine similarity.PPMI uses positive co-occurrence information; SVD reduces PPMI embeddings; SGNS predicts co-occurrence relationships.
- 2.1.2 SVD: SVD reduces PPMI embeddings to a low-dimensional representation, improving robustness through dimensionality reduction as regularization.The representation uses a truncated singular value decomposition with eigenvalue weighting parameter γ ∈ [0, 1].
- 2.2 Datasets: Six historical datasets from Google N-Grams and COHA cover multiple languages and sources, balancing scale against corpus representativeness.Google N-Grams are extremely large but contain sampling artifacts, whereas COHA is genre-balanced and representative of American English over 200 years.
- 2.3 Alignment: Low-dimensional embeddings are aligned across time with orthogonal Procrustes, preserving cosine similarities while resolving arbitrary rotational differences.The best rotational alignment is obtained efficiently using SVD.
- 2.4 Measures of semantic change: Semantic change is quantified either by temporal shifts in pairwise cosine similarity or by cosine-distance displacement of an individual word across periods.Spearman correlation against time assesses the magnitude and significance of pairwise shifts without assuming a particular curve shape.
3 Comparison of different approaches
The study compares embedding methods on synchronic similarity and diachronic semantic-change tasks, using both attested shifts and discovered changes. SVD leads on synchronic accuracy, while SGNS is strongest for discovering shifts and SVD is effective for detecting subtle changes, with method-specific tradeoffs.
- Synchronic accuracy: SVD achieved the highest MEN similarity correlation (ρ = 0.739), ahead of PPMI (ρ = 0.687) and SGNS (ρ = 0.649).The benchmark measured agreement with human word-similarity judgments within time periods.
- Diachronic validity: The diachronic evaluation tested detection of 28 known shifts and discovery of each method’s top-10 most-changing words.Known shifts were evaluated against historical literature and corpora, while discovered shifts were judged for genuineness.
- Diachronic validity: All methods performed almost perfectly at capturing the correct direction of independently attested semantic shifts.The task compared whether word pairs moved closer or farther apart during predetermined periods using ENGALL and COHA data.
- Diachronic validity: The benchmark table reports SGNS as best on ENGALL detection and SVD as best on COHA detection.These values use the paper’s improved and corrected experimental protocol, whose absolute scores are lower than earlier versions.
- Diachronic validity: 70% of SGNS’s top-10 discoveries were genuine semantic shifts, compared with 40% for SVD and 10% for PPMI.PPMI produced borderline cases and rare-event artifacts, whereas SVD was sensitive to corpus artifacts despite finding genuine shifts.
- Methodological recommendations: SVD and SGNS are both reasonable choices with different tradeoffs: SVD favors sensitivity, whereas SGNS is more robust for discovery and visualization.SVD is more effective for subtle shifts but can produce corpus-artifact false discoveries; SGNS is less sensitive on small-dataset detection.
4 Statistical laws of semantic change
The analysis relates semantic change to word frequency and polysemy using diachronic embeddings and regression models across languages and corpora. It finds that frequent words change more slowly, whereas polysemous words change faster after controlling for frequency.
- Overview: The study analyzes how semantic-change rates depend on word frequency and polysemy using diachronic embeddings across languages and time periods.The analysis uses SGNS embeddings and examines one data point per word for each pair of consecutive decades.
- Model: The model controls for temporal trends and repeated measurements of the same word while estimating frequency and polysemy effects.It uses a linear mixed model with random intercepts per word and fixed effects for frequency, polysemy, and decade.
- Frequency: Frequent words change more slowly: semantic-change rates scale with a negative power of frequency, with βf ∈ [−1.24, −0.30] across languages and datasets.The logarithm of frequency has a significant and substantial negative effect on semantic-change rates.
- Polysemy: Polysemy is estimated from contextual diversity in a co-occurrence network, using a measure based on local clustering among a word’s neighbors.Words appearing in disjoint or unrelated contexts receive lower clustering coefficients and higher polysemy scores.
- Limitations: The polysemy measure is biased toward rating contextually diverse discourse-function words as highly polysemous, requiring care when interpreting results.Its intrinsic correlation with frequency is also substantial, with Pearson correlations in the range 0.45 < r < 0.8.
- Polysemy: Polysemous words change faster after controlling for frequency, with a power-law relationship and βd ∈ [0.08, 0.53].The positive polysemy effect reverses the expectation based on polysemy’s positive correlation with frequency.
5 Discussion
The paper argues that distributional methods support robust, large-scale analysis of semantic change and reveal two statistical laws linking frequency and polysemy to change rates. These laws motivate frequency-aware null models and highlight polysemy’s role, while proposed causal mechanisms remain unconfirmed.
- The study extends prior distributional approaches by rigorously comparing methods for quantifying semantic change and proposing new statistical laws.The authors frame this as a robust methodology for future historical-semantic research.
- The analysis spans six corpora, four languages, and two centuries, with English conditions included in the cross-linguistic regression analysis.The reported polysemy-score distributions also vary substantially across languages.
- The law of conformity states that frequent words change more slowly, making frequency a required null model for future semantic-change studies.The authors caution that apparent interactions may primarily reflect underlying frequency effects.
- The law of innovation states that polysemous words change more quickly, suggesting that polysemy may lead to semantic change.This result addresses a question that has concerned linguists for more than a century.
- The proposed laws are compatible with several causal mechanisms, but the results cannot confirm those causal links.The discussion mentions learning-based mechanisms and sociocultural conformity bias as possibilities.
A Hyperparameter and pre-processing details
The appendix specifies vocabulary filtering, context construction, smoothing, subsampling, and implementation choices for the PPMI, SVD, and SGNS models.
- The Google models use the top 100,000 words by average frequency, whereas COHA uses the top 50,000.Words below yearly thresholds of 500 Google occurrences or 100 COHA occurrences are discarded during learning.
- All methods use context distribution smoothing with parameter 0.75, while SGNS and PPMI use α = log(5) and SVD uses α = 0.The SVD setting was chosen because it improved results.
- SGNS on Google data applies subsampling with 10^-5 f(w_i) and downsamples larger years to at most 10^9 tokens.COHA data receives no such subsampling.
- The context vocabulary is generally identical to the target vocabulary, but PPMI retains low-frequency context words because larger contexts substantially improve its results.Other embedding approaches drop those terms for computational efficiency.
- SGNS uses the implementation from Levy et al., while the PPMI and SVD implementations are released with the associated code package.
B Visualization algorithm
The visualization procedure places historical target-word embeddings into a shared modern neighborhood so semantic movement can be inspected across time.
- The procedure first forms the union of a target word’s k nearest neighbors across all required time points.
- It computes a t-SNE embedding for these words at the most recent time point.
- For earlier time points, background embeddings remain fixed while only the target word’s position is optimized.The target is initialized at the centroid of its k′-nearest neighbors, which the authors found effective.
- Because background words stay in their modern positions, the visualization approximates a world in which only the target word moves.The authors note that this approximation is necessary because all words move in reality.
C Regression analysis details
The regression analysis normalizes contextual diversity across years, uses a buffer period, and models feature and time effects with word-level random effects.
- Contextual-diversity scores are normalized within each year by subtracting the yearly median.This addresses changes in yearly medians caused by corpus sample sizes and related factors.
- The analysis excludes the first four decades as a buffer for initializing historical word vectors and measures change only afterward.
- The model includes fixed effects for frequency, contextual diversity, and time, plus random effects for word identity.Part-of-speech tags are omitted because contextual diversity is collinear with them.
- Linear mixed models are fitted with restricted maximum likelihood, and significance scores use Wald’s z-tests.
D Revisions to the methodology for “detecting known shifts” (Table 3)
The revised methodology for detecting known semantic shifts relaxes word-frequency inclusion criteria, requires at least five time points for correlations, and corrects an SGNS dating error for “gay.”
- Cosine similarities are now computed when both words exceed the embedding-specific minimum count, replacing the earlier stricter and insufficiently documented frequency cutoff.The revised thresholds are 100 occurrences for COHA and 500 for the ENGALL corpus, producing lower overall scores.
- At least 5 time points are now required before computing Spearman correlations, adding a minimum robustness condition.
- The revised methodology corrects the SGNS model’s previously incorrect start date for the semantic shift of “gay.”
- Not all word pairs listed in Table 2 are evaluated in every setting because some corpora, including COHA, lack enough samples.
- A replication script for the revised Table 3 results is available in the associated GitHub repository.