PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Word Embeddings

Robert Bamler, Stephan Mandt

arXiv:1702.08359v2stat.MLcs.LG

TL;DR

The paper addresses the difficulty of tracking changing word meanings when static embeddings assume constant semantics and time-slice methods cannot reliably share information across periods. It introduces a probabilistic dynamic skip-gram model with diffusion-based embedding trajectories and scalable filtering and smoothing inference. Experiments across three corpora show smoother, more interpretable trajectories and better held-out predictive likelihoods than separately trained static models.

  • Problem

    Static embeddings assume constant word meaning, while separate time-bin training can misalign embeddings and overfit when each period has little data.

  • Method

    The model combines Bayesian skip-gram with a latent diffusion process over word and context embeddings, trained jointly using scalable filtering and smoothing inference.

  • Results

    Across three corpora, the model produces smooth, interpretable embedding trajectories and better held-out predictive likelihoods than separately trained static models.

  • Takeaways & Limitations

    Dynamic skip-gram can track semantic change and support data mining, anomaly detection, and historical or social-scientific study of language evolution.

  • Takeaways & Limitations

    The diffusion constant must be chosen for the application’s semantic time scale, which may vary across corpora and shifts.

Abstract

from arXiv · show

We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.

1. Introduction

Language changes over time, but static word embeddings assume each word has one meaning across an entire corpus. The paper introduces a dynamic model that learns continuous semantic trajectories from sequential text while sharing information across time.

  • Semantic shifts arise from cultural, technological, and political change, motivating models that detect changing word meaning and usage.
  • Static embeddings assume that a word’s meaning remains constant across the entire corpus, limiting their use for historical or streaming text.
  • Time-slice training can produce inconsistently aligned embeddings and overfit when fine-grained bins contain too little data.
  • The proposed probabilistic state-space model lets word and context embeddings evolve through a diffusion process, enabling end-to-end training and information sharing across times.
  • Two scalable variational inference algorithms—filtering and smoothing—learn dynamic embeddings that generalize better to held-out data.
  • Across three massive corpora, the approach produces smooth trajectories and automatically identifies words with the largest semantic changes.

2. Related Work

Earlier approaches either model topics without semantic relations or train static embeddings separately in time bins. Dynamic skip-gram instead connects Bayesian skip-gram models across time to share statistical strength.

  • Latent time-series models such as dynamic topic models track evolving topics, but bag-of-words representations treat words as symbols without modeling semantic relations.
  • Bayesian and standard skip-gram models provide scalable word embeddings but do not allow embedding vectors to change over time.
  • The proposed model connects T copies of Bayesian skip-gram through a latent time-series prior on the embeddings.
  • Prior semantic-change methods slice corpora into time bins, fit separate models, and analyze the resulting embeddings in post-processing.
  • Separate time-bin training cannot share statistical strength across time and may require sufficiently large bins.

3. Model

Dynamic skip-gram generalizes Bayesian word2vec to sequential text by assigning word and context embeddings at each time and coupling them through a diffusion prior. This prior encourages smooth trajectories while sharing information across observations.

  • Dynamic skip-gram is a probabilistic, end-to-end generalization of word2vec for sequential text, with embeddings that continuously drift over time.
  • At each time step, the model learns word and context vectors from word-context statistics within a context window.
  • The Bayesian skip-gram component assigns probabilities to word-context pairs using the inner product of word and context embeddings passed through a sigmoid.
  • Positive examples are observed neighboring word pairs, while negative examples are constructed from rejected word-word pairs in the generative process.
  • A Kalman-filter prior governs embedding evolution, allowing vectors to drift while sharing information across all times.
  • The diffusion prior enforces smooth variation across time, supports unambiguous word association, and helps fit settings with small per-time datasets when total data are large.

4. Inference

The paper presents two scalable variational inference strategies for its dynamic embedding model: filtering for sequential data and smoothing for joint inference over complete document sequences. Smoothing uses future and past evidence to produce smoother trajectories and typically higher likelihoods, while structured variational inference is made computationally efficient.

  • Filtering: Filtering updates the variational distribution sequentially using current evidence and the approximate posterior from the previous time step.It is designed for settings where data arrive over time and future observations are unavailable.
  • Filtering: Filtering uses a time-factorized variational distribution with fully factorized, diagonal covariance matrices at each time.This is a mean-field approximation across embedding dimensions and vocabulary factors at each step.
  • Smoothing: Smoothing conditions inference on the entire observation sequence, using both future and past evidence to obtain smoother trajectories and typically higher likelihoods than filtering.Unlike filtering, smoothing requires the full sequence of documents in advance.
  • Smoothing: Joint smoothing inference uses a structured Gaussian variational distribution with a tridiagonal precision matrix to capture correlations across time.The variational parameters include the time-indexed means and entries of the precision matrix.
  • Efficient inference: An efficient Cholesky-based gradient algorithm reduces the structured inference update cost from Θ(T^2) time and memory to Θ(T).The method parameterizes the precision matrix through a bidiagonal Cholesky factor.

5. Experiments

Experiments on three time-stamped corpora show that dynamic skip-gram models produce smoother trajectories and higher held-out predictive likelihoods than static baselines, especially with sparse data per time step.

  • Algorithms and Baselines: The experiments compare DSG-F and DSG-S with static baselines SGI and SGP across Google Books, State of the Union, and Twitter corpora.The corpora cover different combinations of time span and data volume per time step.
  • Qualitative results: The dynamic models produce smoother trajectories and automatically detect words undergoing substantial semantic change.The evaluation follows embedding movement and semantic shifts across all three corpora.
  • Trajectory smoothness: DSG-F shows directed, gradual movement from 1998 onward, whereas SGI and SGP show little directed motion after the first year.The comparison uses distances between word vectors in 1998 and subsequent years.
  • Trajectory smoothness: DSG-F and DSG-S produce less noisy trajectories than SGI and SGP, with the clearest difference on the smaller SoU and Twitter corpora.The static baselines predict no correlation for selected word pairs in these smaller corpora, while the dynamic models share information across time.
  • Quantitative results: The dynamic models achieve higher held-out predictive likelihoods than static models, with the largest improvement on the smaller SoU and Twitter corpora.DSG-S outperforms DSG-F because smoothing shares information in both time directions.

6. Conclusions

The dynamic skip-gram model combines word2vec with a latent continuous time series to model smoothly changing embeddings. Filtering and smoothing improve held-out prediction, especially when each time step contains little data.

  • Conclusion: The dynamic skip-gram model is a Bayesian probabilistic model combining word2vec with a latent continuous time series.It supports time-varying word and context embeddings.
  • Conclusion: Dynamic skip-gram filtering and smoothing produce smoothly changing embedding vectors and better predict word-context statistics at held-out time steps.Filtering conditions only on past observations, whereas smoothing uses all data.
  • Conclusion: The benefits are greatest when individual time steps contain little data and static embedding models are difficult to fit.The paper also identifies applications in streaming-text mining, anomaly detection, and historical language analysis.

arXiv:1702.08359v2 [stat.ML] 17 Jul 2017

The experiments use fixed vocabulary, embedding, optimization, sampling, and diffusion settings for the three corpora and inference procedures.

  • Hyperparameters: The experiments use vocabulary size L = 104, smoothing batch size L′ = 103, and embedding dimensions d = 100 or d = 200.Google Books uses d = 200, while SoU and Twitter use d = 100.
  • Sampling: The context window is cmax = 4, the negative-to-positive ratio is η = 1, and the negative-sampling context exponent is γ = 0.75.These parameters govern the construction of positive and negative word-context examples.
  • Optimization: The global prior variance is σ^2_0 = 1, while learning rates are 10^-2 for filtering and the smoothing minibatch phase and 10^-3 afterward.Adam decay rates and regularization are also specified for each inference procedure.

1. Dimensionality Reduction in Figure 1

Figure 1 projects fitted word embeddings into two dimensions with dynamic t-SNE, balancing local neighborhood preservation against temporal alignment. The alignment penalty is annealed toward zero to avoid biasing trajectories toward smoothness while retaining meaningful cross-time comparisons.

  • Projection method: Dynamic t-SNE projects fitted embeddings from R^d into two dimensions while preserving nearest-neighbor relations at each time step.It is a non-parametric dimensionality-reduction method for sequential data.
  • Temporal alignment: A quadratic penalty with prefactor λ aligns projections at neighboring time steps by penalizing sudden movements.The penalty supplements each time step’s local projection objective.
  • Trade-off: λ trades off locally good projections against smooth projections, but λ = 0 leaves independent rotations that make projected trajectories meaningless.The optimization is invariant under independent rotations at each time when λ is zero.
  • Annealing schedule: λ starts at 0.01 and decreases by 5% per training step for 100 steps, reaching approximately 6×10^-6.Exponential annealing reduces smoothness bias while maintaining temporal alignment during optimization.

2. Hyperparemeters and Construction of n±

The experiments use corpus-specific embedding dimensions and deterministic positive and negative count matrices that approximate word2vec’s stochastic context-window sampling. These preprocessing choices retain distance-dependent context weighting while avoiding explicit negative-count matrices.

  • Hyperparameters: Embedding dimensions match prior settings for Google Books but are reduced for SoU and Twitter to avoid overfitting smaller datasets.The context-window size and dimension for Google Books follow Kim et al. (2014).
  • Count construction: Positive and negative count matrices are constructed deterministically during preprocessing rather than sampled during each word2vec update.The construction is designed to resemble word2vec’s stochastic approach as closely as possible.
  • Positive counts: For a word pair separated by k words, the positive count increases by max(0, 1 − k/cmax) when 0 ≤ k ≤ cmax.Nearby words therefore receive larger positive counts than more distant words within the context window.
  • Negative counts: Negative examples use η independent draws from a distribution chosen to match word2vec’s expected negative word-context counts.The implementation sets γ = 0.75 and η = 1, while retaining distributions instead of full negative matrices.

3. Skip-gram Filtering Algorithm

Algorithm 1 provides pseudocode for skip-gram filtering, the inference procedure described for the dynamic model’s sequential setting.

  • Algorithm overview: Algorithm 1 summarizes skip-gram filtering, which is described in section 4 of the main text.The supplied passage identifies the algorithm but does not provide its computational steps.

4. Skip-gram Smoothing Algorithm

Skip-gram smoothing uses structured Gaussian variational distributions and stochastic optimization to infer temporally correlated embedding trajectories. Efficient gradient computation and vocabulary minibatching reduce the cost of joint inference across all time steps.

  • Variational distribution: The smoothing variational distribution uses a bidiagonal Cholesky factor, yielding a Gaussian with tridiagonal precision and correlations across time steps.Setting the off-diagonal parameters to zero recovers mean-field; nonzero values permit dense covariance across time.
  • Optimization: Stochastic gradient ascent maximizes the ELBO, using analytic entropy terms and reparameterization-trick samples for intractable likelihood gradients.The implementation uses one sample per training step and Adam updates.
  • Vocabulary sampling: Vocabulary minibatching reduces a training step’s complexity from L^2 to (L′)^2 before full-batch refinement.The method pretrains for 5000 minibatch steps and then trains for 1000 full-batch steps.
  • Vocabulary sampling: Uniformly sampled word and context minibatches update only corresponding variational parameters, with likelihood contributions rescaled by L/L′.The prior and variational distribution factorize over vocabulary, although the likelihood does not.
  • Constraint handling: Mirror ascent keeps the Cholesky parameters νu,t positive, preserving positive definiteness of the variational precision matrix.Positive and negative optimizer steps respectively increase and decrease νu,t without crossing zero.
Loading 1702.08359v2…