Source-linked AI summary
Dynamic Word Embeddings for Evolving Semantic Discovery
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, Hui Xiong
TL;DR
Changing word associations make static embeddings inadequate for studying language evolution. The paper jointly learns time-aware embeddings with temporal alignment regularization from New York Times data, and reports stronger semantic accuracy and alignment quality than prior temporal approaches.
Problem
Static word representation methods generally ignore temporal factors, while independently learned time-slice embeddings create an alignment problem that complicates cross-time comparison.
Method
The paper jointly learns low-dimensional word embeddings for chronologically sliced corpora and regularizes changes across time to align the representations.
Results
The method consistently outperforms state-of-the-art temporal embedding approaches on semantic accuracy and alignment quality in qualitative and quantitative evaluations.
Takeaways & Limitations
Joint temporal modeling provides more interpretable, data-efficient, and reliably aligned embeddings for discovering evolving word semantics.
Takeaways & Limitations
The scalable block coordinate descent optimization has a drawback: it lacks convergence guarantees, even for convex optimization.
Abstract
from arXiv · showhide
Word evolution refers to the changing meanings and associations of words throughout time, as a byproduct of human language evolution. By studying word evolution, we can infer social trends and language constructs over different periods of human history. However, traditional techniques such as word representation learning do not adequately capture the evolving language structure and vocabulary. In this paper, we develop a dynamic statistical model to learn time-aware word vector representation. We propose a model that simultaneously learns time-aware embeddings and solves the resulting "alignment problem". This model is trained on a crawled NYTimes dataset. Additionally, we develop multiple intuitive evaluation strategies of temporal word embeddings. Our qualitative and quantitative tests indicate that our method not only reliably captures this evolution over time, but also consistently outperforms state-of-the-art temporal embedding approaches on both semantic accuracy and alignment quality.
1 INTRODUCTION
The paper models word meanings as time-dependent embeddings to capture evolving associations, while jointly addressing cross-time alignment. It evaluates the approach on New York Times data with qualitative and quantitative tests.
- Motivation: Word semantics change over time, motivating embeddings that capture time-aware meanings and associations.The paper connects this goal to applications such as time-aware knowledge extraction and text mining.
- Motivation: Traditional word embeddings generally treat words as static and therefore do not model temporal semantic change.Methods such as word2vec and GloVe improve word representation learning but usually omit temporal factors.
- Problem: The alignment problem arises because independently learned embeddings can occupy differently rotated latent spaces across time slices.Training objectives can be rotation-invariant, making direct cross-time comparison unreliable.
- Approach: The proposed method jointly learns embeddings across time and uses regularization to smooth changes, avoiding a separate alignment stage.This approach shares information across time slices for much of the vocabulary.
- Results: Experiments report better results than two-step alignment methods and robustness to sparse or missing words in time slices.The paper presents this robustness as a practical advantage of sharing information across time.
- Evaluation: The model is trained on New York Times articles from 1990 to 2016 and evaluated with qualitative and quantitative metrics.The evaluation includes semantic accuracy, cross-time alignment, word-vector trajectories, associated-word discovery, and embedding norms.
2 METHODOLOGY
The methodology represents each word in each time period with a low-dimensional vector learned from time-sliced corpora. A joint factorization objective combines data fidelity with temporal smoothing to align embeddings while permitting change.
- Temporal corpus: The corpus is divided into chronologically ordered time slices, which may span months, years, or decades and have unequal lengths.The vocabulary includes words appearing at any time, including words absent from particular slices.
- Temporal representation: For every vocabulary word and time period, the model seeks a dense low-dimensional vector representation.The embedding dimension d is much smaller than the vocabulary size and is typically 50–200.
- Static embedding basis: Static embeddings rely on the observation that semantically similar words tend to have similar neighboring words in a corpus.Neighbor structure is captured through word co-occurrence within a local context window.
- Temporal embedding construction: The model uses time-specific PPMI matrices and low-rank factorization to derive embeddings from co-occurrence statistics.The PPMI construction uses a context window of size 5 in this paper and improves numerical stability over raw PMI for weak associations.
- Alignment problem: Independent factorization is insufficient for alignment because orthogonal rotations leave the reconstruction error unchanged.Thus, embeddings for different time slices may not share a common latent coordinate system.
- Joint model: Joint alignment uses information from all time slices and can preserve an absent word’s representation when neighboring slices provide related evidence.The paper contrasts this with pairwise procedures that may propagate alignment errors or collapse absent-word vectors toward zero.
- Joint model: The joint optimization combines low-rank data fidelity, embedding regularization, and a temporal smoothing penalty that encourages alignment.The parameter τ controls the permitted rate of change: τ = 0 gives no alignment, while τ →∞ yields a static embedding.
3 OPTIMIZATION
The optimization addresses scalability by decomposing time-indexed objectives and updating smaller variable blocks, while recognizing trade-offs among solution methods.
- Scalable optimization: Alternating minimization decomposes the objective across time to avoid storing all sparse PPMI matrices when vocabulary and time dimensions are large.It solves for U (t) at each time step.
- Optimization choices: The fixed-time objective is quartic in U (t), so it cannot be minimized analytically and requires iterative first-order optimization such as gradient descent.The gradient of the first term is given as −2Y(t)U (t) + 2U (t)U (t)^TU (t).
- Scalable optimization: Introducing W (t) converts each U (t) update into a ridge regression solvable in one step, with row-block updates reducing complexity independently of V.Forming and solving the updates requires O(Vd^2 + nnz(Y(t))d) and O(d^3), while row blocks reduce formation cost to depend on block size b.
- Optimization choices: Block coordinate descent minimizes one U (t) or W (t) block at a time, with smaller blocks available to maintain scalability.Its main appeal is scalability, but it lacks convergence guarantees even for convex optimization.
- Optimization choices: SGD decomposes the objective into smaller terms but can suffer from slow progress and difficult step-size tuning, making it slower in practice for matrix factorization.The choice of optimization method is agnostic to the model if it successfully solves the objective.
4 EXPERIMENTAL DATASET AND SETUP
The experiments use a yearly, 27-slice New York Times corpus spanning 1990 to July 2016, with article text and metadata across 59 sections.
- Dataset: The dataset contains 99,872 New York Times articles published between January 1990 and July 2016.The corpus includes article text plus title, author, release date, and section label metadata.
- Temporal setup: The corpus is divided into T = 27 yearly time partitions for temporal embedding experiments.These yearly slices cover the stated publication period.
- Dataset: The collected metadata includes 59 article sections, including Business, Sports, and Technology.The section label is one of the recorded metadata fields.
5 QUALITATIVE EVALUATION
The qualitative evaluation uses temporal embeddings to visualize semantic shifts, search cross-time equivalences, and track concept popularity in news.
- 5.1 Trajectory visualization: Temporal embedding trajectories capture semantic shifts while keeping same-meaning words across years geometrically close without separate alignment optimization.Figure 1 includes apple, amazon, obama, and trump across the 27-year period.
- 5.1 Trajectory visualization: Apple shifts from fruit and dessert associations toward technology, with a 1994 discussion spike and recovery in the early 2000s.The passage links the spike to CEO replacement and collaboration with IBM, and the later recovery to Steve Jobs.
- 5.2 Equivalence searching: Cross-time nearest-word searches reveal technology substitutions, changing news sources, and music-consumption replacements.Examples include iphone, twitter, and mp3, with Napster and iTunes appearing at specific points in the trajectory.
- 5.2 Equivalence searching: The president query retrieves the correct closest person for every year, while the mayor query is mostly consistent but has no relevant word in 2006.Cuomo is partially relevant in 2011 because he was New York’s governor.
- 5.2 Equivalence searching: Alignment searches recover many correct ATP No.1 tennis champions despite the absence of role titles, and all retrieved names are famous players from the relevant periods.The paper presents this as a qualitative indication of alignment quality before the more exhaustive quantitative study.
- 5.3 Popularity determination: Embedding norms provide smoother popularity trends than noisy frequencies, tracking presidential terms and temporary events such as Enron and al-Qaeda.The norms show Trump’s 2016 popularity surpassing Clinton’s news coverage, while frequency can spike misleadingly for Enron.
6 QUANTITATIVE EVALUATION
The evaluation tests semantic categorization and cross-year alignment of temporal embeddings, finding that DW2V consistently outperforms the baselines on clustering and alignment quality.
- Evaluation setup: DW2V is evaluated against SW2V, TW2V, and AW2V with embedding dimension d = 50.SW2V ignores time, TW2V applies local transformations, and AW2V aligns embeddings between adjacent slices.
- Semantic similarity: The semantic categorization test uses New York Times section labels as ground truth without using those labels during embedding learning.The dataset includes examples such as amazon shifting from World in 1995 to Technology in 2012.
- Semantic similarity: The clustering dataset contains 1888 word-year-section triplets across 11 sections, filtered for strong section associations.Each section contributes at most 200 qualified words, and triplets require section strength of at least 35%.
- Semantic similarity: Clustering uses spherical k-means with cosine distance and evaluates results using NMI and Fβ-measure.The tests use K = 10, 15, and 20 clusters; Fβ weights recall more heavily with β = 5.
- Semantic similarity: DW2V consistently outperforms the baselines across all tested K values, indicating semantic-shift capture and strong alignment across years.AW2V also performs well, while TW2V’s local alignment is insufficient for high overall alignment quality.
- Alignment quality: The alignment evaluation measures ranking and top-K retrieval using MRR and Mean Precision@K.MRR uses the reciprocal rank of the correct answer, assigning zero when it is absent from the top 10.
- Alignment quality: DW2V outperforms competing methods on cross-year equivalence queries, sometimes by an order of magnitude.Testset 1 uses recorded role equivalences, while testset 2 covers longer-range human-generated associations; SW2V degrades sharply on the latter.
- Robustness: Under extreme co-occurrence attacks leaving 1% or 0.1% of co-occurrences, DW2V remains steady while the comparison method degrades sharply.The comparison is reported for r = 1, 0.1, 0.01, and 0.001.
7 RELATED WORK
Prior work studies temporal language through topic models, word counts, and temporal word embeddings, while embedding research spans co-occurrence, matrix-factorization, and neural approaches.
- Temporal effects in natural language processing: Temporal NLP studies use topic modeling and time-stamped journals to identify emerging themes and viewpoints.Other work uses word counts to study cultural phenomena, teen behavior, and influenza incidents.
- Word embedding learning: Word embeddings evolved from co-occurrence and matrix-factorization methods to neural approaches such as GloVE and word2vec.These low-dimensional embeddings improved performance in document clustering, LDA, and word similarity.
- Temporal word embeddings: Related temporal embedding methods commonly use a two-step process of learning separate time-slice embeddings and aligning them afterward.Prior studies also evaluate semantic shift through neighbor-angle tracking, synthetic shifts, and timestamp prediction.
8 CONCLUSION
The paper formulates evolving word semantics as dynamic embedding learning and proposes jointly learned, aligned representations evaluated through qualitative and quantitative tests.
- Conclusion: The proposed model jointly learns time-aware embeddings and aligns them across time using scalable block coordinate descent.The authors report benefits including higher interpretability, better quality with less data, and more reliable across-time querying.
- Conclusion: Qualitative and quantitative evaluations show that the dynamic embedding method performs favorably against other temporal embedding approaches.The evaluations target evolving word semantics and across-time alignment.