Source-linked AI summary
Continuous Time Dynamic Topic Models
Chong Wang, David Blei, David Heckerman
TL;DR
The paper addresses the difficulty of modeling evolving topics when discrete-time dynamic topic models require a computationally costly choice of temporal discretization. It introduces a Brownian-motion cDTM with sparse variational inference, and demonstrates faster model comparison and evaluation on two real-world datasets.
Problem
The dDTM requires discretized time, while finer granularity increases variational-inference complexity and can limit analysis at the appropriate time scale.
Method
The cDTM uses Brownian motion for continuous-time topic evolution and sparse variational inference that exploits the sparsity of observed words.
Results
The cDTM supports fast model comparison across granularities and was evaluated using predictive likelihood and time-stamp prediction accuracy on two real-world datasets.
Takeaways & Limitations
The cDTM makes temporal granularity a modeling choice rather than one governed by computational concerns.
Abstract
from arXiv · showhide
In this paper, we develop the continuous time dynamic topic model (cDTM). The cDTM is a dynamic topic model that uses Brownian motion to model the latent topics through a sequential collection of documents, where a "topic" is a pattern of word use that we expect to evolve over the course of the collection. We derive an efficient variational approximate inference algorithm that takes advantage of the sparsity of observations in text, a property that lets us easily handle many time points. In contrast to the cDTM, the original discrete-time dynamic topic model (dDTM) requires that time be discretized. Moreover, the complexity of variational inference for the dDTM grows quickly as time granularity increases, a drawback which limits fine-grained discretization. We demonstrate the cDTM on two news corpora, reporting both predictive perplexity and the novel task of time stamp prediction.
1 Introduction
The paper introduces the cDTM to model topics that evolve through time, addressing the exchangeability and discretization assumptions of earlier topic models. Its sparse inference enables fitting models at varying temporal granularities, which the authors demonstrate on news corpora.
- Most topic models assume documents are exchangeable, although collections such as news and scientific journals evolve over time.
- The dDTM discretizes time, creating a trade-off between coarse temporal assumptions and rapidly increasing inference complexity at fine resolutions.Discretization affects memory requirements and computational complexity, limiting the resolution at which the model can be fit.
- The cDTM replaces the dDTM’s discrete state space model with Brownian motion, allowing topic evolution to be modeled in continuous time.Only the resolution of document time stamps is discretized.
- Sparse variational inference exploits the sparsity of text, making it possible to fit cDTMs at varying granularities.The paper contrasts this capability with the dDTM’s computational limitations and applies the models to two news corpora.
- Unlike TOT and DMM, which keep topics constant and use time information to discover them, the cDTM models topics themselves as evolving.
2 Continuous time dynamic topic models
The cDTM represents topics as distributions over a fixed vocabulary and models their natural parameters with Brownian motion. By avoiding representation of unobserved intermediate time points, it makes temporal granularity a modeling choice rather than a computational constraint.
- A topic is represented as a distribution over the collection’s fixed vocabulary, with multinomial natural parameters given by log probabilities.
- The cDTM models topic evolution with Brownian motion, with variance determined by the lag between observations.Probabilities at discrete steps between observed documents need not be represented.
- The dDTM explicitly represents topics at discrete time ticks, so fine-grained timelines require high posterior-inference memory even when observations are sparse.
- The cDTM avoids representing log probabilities at unobserved intermediate ticks, enabling sparse inference and arbitrary time granularity.A dDTM can be obtained by measuring document time stamps at the desired granularity.
- The cDTM is the limiting process of the dDTM as the per-tick variance approaches zero.
3 Sparse variational inference
Sparse variational inference avoids explicitly representing unobserved vocabulary terms, reducing memory from dense O(VT) scaling to the number of unique observed terms across time points. This enables efficient fine-grained analysis while retaining the cDTM’s variational inference framework.
- Variational inference: Posterior inference is intractable, so variational methods approximate the true posterior by optimizing distributions over topic proportions, word-topic assignments, and topic sequences.The variational parameters include Dirichlet γt, multinomial φ, and β̂ variables used as observations by a variational Kalman filter.
- Sparse representation: The sparse algorithm omits β̂t,w whenever word w has no observations at time t, because its posterior is determined by observations of other words.This preserves the relevant belief-propagation structure while avoiding explicit variables for unobserved terms.
- Forward-backward inference: The forward-backward procedure computes Gaussian variational distributions for each term, with unobserved terms carrying forward their previous mean and observed terms receiving updated Gaussian observations.The algorithm then optimizes only variational observations associated with observed terms.
- Complexity: Memory scales as O(∑w δt,w), the total number of unique observed terms across time points, rather than O(VT) for dense inference.Conjugate gradient optimization updates only variational observations with δt,w = 1.
- Empirical efficiency: With 6243 Science time points, the 10-topic cDTM requires 0.8G memory versus 2.3G for the dDTM, while Science has sparsity 0.65.The finer issue-level resolution is therefore substantially more memory-efficient under the cDTM.
4 Experiments
Experiments evaluate cDTM on two time-stamped news corpora using predictive perplexity and time stamp prediction. Results show that sparse inference supports efficient granularity comparisons, while suitable topic complexity and temporal resolution depend on the data.
- 4.1 News Corpora: The experiments use AP election news with hourly timestamps and Election 08 summaries with daily timestamps.AP contains 1,342 documents; Election 08 contains 1,040 summaries.
- 4.2 Per-Word Predictive Perplexity: The evaluation measures per-word predictive perplexity using previous time indices and averages it over the timeline.Lower perplexity is better, and each document is predicted exactly once across granularities.
- 4.2 Per-Word Predictive Perplexity: For AP, sparse inference fits granularities efficiently, day and week models perform similarly, and 5- and 10-topic models perform better.The result compares temporal granularities and topic counts on the AP data.
- 4.2 Per-Word Predictive Perplexity: For Election 08, the 1-topic model performs best because the summaries are very short, while finer granularity does not necessarily improve prediction.Performance varies with the amount of data supported at each time point.
- 4.3 Time Stamp Prediction: Time stamp prediction uses an 80% training and 20% testing split, evaluates average absolute error, and compares flat with hierarchical prediction.The hierarchical approach successively narrows predictions from month to week to day when needed.
- 4.3 Time Stamp Prediction: The hierarchical approach performs at least as well as the flat approach, while larger granularities work better on these small data sets.The hour model for AP and day model for Election 08 perform worse; parameter v is shared across models and is left for future inference.
- 4.4 Example Topics: Example Election 08 topics shift from general issues such as healthcare in 2007 toward candidate-focused topics that change faster in 2008.Topics are sampled every two months using the week model.
5 Conclusions
The paper develops cDTM, using Brownian motion for continuous-time topic evolution and sparse variational inference for fast model comparison. It demonstrates the model through predictive likelihood and time stamp prediction on two real-world data sets, and proposes bounded-variance OU dynamics as future work.
- The paper develops cDTM using Brownian motion to model continuous-time topic evolution.
- Sparse variational inference is the cDTM's main advantage for fast model comparison.
- The model is demonstrated with predictive likelihood and time stamp prediction accuracy on two real-world data sets.
- Future work will explore the Ornstein-Uhlenbeck model, which allows bounded variance.