Source-linked AI summary
Timeline: A Dynamic Hierarchical Dirichlet Process Model for Recovering Birth/Death and Evolution of Topics in Text Stream
Amr Ahmed, Eric P. Xing
TL;DR
Existing topic models do not generally capture the joint evolution of topic number, word distributions, and popularity in temporal document streams. The paper introduces iDTM, an unbounded dynamic topic model with topic birth and death and Markovian topic evolution, and evaluates it on simulated data and NIPS proceedings, where it recovers simulated topic distributions and lifespans.
Problem
Temporal document streams can have evolving topic numbers, topic distributions, and topic popularity, while existing models capture only some of these aspects.
Method
iDTM models unbounded topics that can be born or die across epochs, with word distributions evolving through a first-order state-space model and popularity evolving through a Δ-order process.
Results
The model was evaluated on simulated data and NIPS proceedings, and recovered the correct topic distributions and lifespans in simulation.
Takeaways & Limitations
iDTM provides timelines of topic birth, evolution, trends, and representative papers for exploring and summarizing the NIPS collection.
Takeaways & Limitations
The inference procedure uses a Laplace approximation for non-conjugate integrals, and the paper identifies sampling all hyperparameters as future work.
Abstract
from arXiv · showhide
Topic models have proven to be a useful tool for discovering latent structures in document collections. However, most document collections often come as temporal streams and thus several aspects of the latent structure such as the number of topics, the topics' distribution and popularity are time-evolving. Several models exist that model the evolution of some but not all of the above aspects. In this paper we introduce infinite dynamic topic models, iDTM, that can accommodate the evolution of all the aforementioned aspects. Our model assumes that documents are organized into epochs, where the documents within each epoch are exchangeable but the order between the documents is maintained across epochs. iDTM allows for unbounded number of topics: topics can die or be born at any epoch, and the representation of each topic can evolve according to a Markovian dynamics. We use iDTM to analyze the birth and evolution of topics in the NIPS community and evaluated the efficacy of our model on both simulated and real datasets with favorable outcome.
1 Introduction
The paper addresses the difficulty of modeling temporal document collections whose topic numbers, distributions, and popularity evolve together. It introduces iDTM to model these dynamics and evaluates the approach on simulated data and NIPS proceedings.
- Motivation: Temporal document collections require models that capture changing topic structure and popularity over time.Existing topic models are useful for organizing large collections, but several latent aspects can evolve in temporal streams.
- Limitations of prior work: Existing models typically evolve only some aspects, such as word distributions and popularity, while keeping the number of topics fixed.Other models capture popularity while fixing topic-word distributions and topic counts, or use restricted trend shapes.
- Contributions: iDTM models topic trends, topic distributions, and the number of topics as evolving quantities.Its formulation allows an unbounded topic count and supports topic birth and death over epochs.
- Contributions: The model uses a dynamic sampling algorithm based on maintaining cached sufficient statistics.
- Evaluation: The paper empirically evaluates and illustrates iDTM on simulated data and NIPS proceedings.It also studies the model’s sensitivity to hyperparameter settings.
2 Settings and Background
The paper models ordered document epochs in which documents are exchangeable within epochs while topic structure evolves across epochs. It builds from temporal Dirichlet-process and hierarchical Dirichlet-process constructions to represent unbounded, multi-topic document collections.
- Data setting: Documents are organized into ordered epochs, with documents and words represented explicitly at each epoch.The model seeks an unbounded number of topics whose word distributions span selected epoch intervals.
- Dirichlet-process background: A Dirichlet process is a distribution over distributions that induces a Chinese restaurant process with recurring parameter values.
- Temporal mixtures: The temporal Dirichlet process mixture model allows components to emerge, persist, or disappear while their parameterizations evolve over time.Its component weights depend on prior epochs through a time-decaying kernel controlled by Δ and λ.
- Limits of single-topic modeling: The recurrent Chinese restaurant process is limited because each document is generated from a single topic.
- Hierarchical mixtures: The hierarchical Dirichlet process instead represents each document with a document-specific mixture over potentially infinite shared topics.Words are assigned to topics through document-level random measures tied to a shared global base measure.
- Limits of static hierarchical modeling: The Chinese restaurant franchise supports multiple topics and an unbounded topic count but does not evolve topic trends or word distributions.
3 Infinite Dynamic Topic Models
iDTM models temporal topic streams with epoch-specific HDPs whose topic availability, popularity, and word distributions evolve over time. Its RCRF foundation links epoch-specific topic menus while allowing topics to be born, die, and persist.
- Model construction: iDTM models each epoch with an epoch-specific HDP and links its base measures through the recurrent Chinese restaurant franchise process.The RCRF ties global topic menus across epochs while preserving epoch-specific document modeling.
- Topic dynamics: Topics can be born or become unavailable over time, with a topic considered dead after being unused for a consecutive ∆ epochs.Newly born topics have zero historic usage, while available but unused topics retain positive historic usage.
- Topic dynamics: A topic’s popularity depends on current usage and usage during the preceding ∆ epochs, implementing time-decayed dependence across epochs.The RCRF figure illustrates a first-order process, while the text describes the general ∆-order construction.
- Word assignments: Within each document, words use the Chinese restaurant franchise assignment mechanism, choosing existing tables or creating a new table and topic.A table represents a group of words sharing a topic; the total probability of choosing a new table remains α.
- Topic evolution: Topic word distributions evolve through a Markovian random walk, and topic parameters are mapped to the simplex before generating words.The model uses φk,t|φk,t−1 ∼ N(φk,t−1, ρI) followed by a logistic transformation and multinomial word generation.
4 A Gibbs Sampling Algorithm
The Gibbs sampler alternates updates for topic assignments, table assignments, and topic parameters in a Markov chain. Because non-conjugate integrals are difficult, the method uses sampling and Laplace-based proposals, while caching reduces transition computation.
- Sampler structure: The sampler constructs a Markov chain over topic assignments, table assignments, and topic parameters, updating each variable conditioned on its Markov blanket.The variables include the topic assigned to each table, the table assigned to each word, and each topic’s epoch-specific parameterization.
- Topic updates: Topic assignments for tables are sampled from a conditional distribution that incorporates local word counts and transition probabilities across neighboring epochs.The transition factor evaluates the likelihood of future table assignments when a table’s topic assignment changes.
- Non-conjugacy: Non-conjugate integrals cannot be computed analytically, so the algorithm replaces them with fresh samples from the relevant distributions.The authors report that deterministic approximation overestimates the integrals and increases the rate of generating new topics.
- Table updates: Table assignments choose between an existing table and a new table, with the new-table probability remaining α after normalizing over available topics.If a word was alone at its table, its table contribution is first removed from the count vector.
- Parameter updates: Topic parameters are sampled with Metropolis-Hastings, and block sampling is preferred because successive epoch parameters are strongly correlated.The proposal is based on a Laplace approximation to the LTR smoother.
- Practical considerations: Caching transition costs reduces the computation from O(K^2∆) to at most O(K∆).The update only recomputes costs involving the old and new topic assignments, with larger improvements possible when assignments change infrequently.
5 Experimental Results
Experiments evaluate iDTM on simulated and NIPS document streams, testing recovery of topic lifespans and evolution, predictive likelihood, and hyperparameter sensitivity. The results show recovery of simulated topic structure, interpretable NIPS timelines, improved predictive LL over DTM and HDP, and sensitivity to key hyperparameters.
- Simulation Results: iDTM recovered the correct topic distributions and lifespans in a simulated 20-epoch collection with eight hand-crafted topics.The simulation used a vocabulary of 16 words and added 5% random noise to each topic’s word distribution.
- NIPS Conference: On NIPS proceedings from 1987–1999, iDTM modeled changes in topic number, word distributions, and topic trends over 13 publication-year epochs.The corpus contained 1,740 documents and a 3,379-word vocabulary after frequency filtering.
- NIPS Conference: Sharp changes in inter-epoch unigram KL divergence coincided with changes in topic number, whereas stable nonzero divergence led to changes in topic words or trends.This behavior distinguishes iDTM from DTM, which changes only topic word distributions and trends.
- NIPS Conference: The NIPS analysis produced timelines of topic births, trends, evolving top words, and representative high-weight papers, including a Kernel topic with a large initial trend in 1996.High-weight papers need not be the most influential papers; later-year examples instead summarize how a topic is addressed over time.
- Quantitative Evaluation: iDTM achieved better predictive held-out log likelihood than DTM and HDP using training and test documents sampled uniformly across epochs.The comparison used a 75% training split and a 25% test split.
- Quantitative Evaluation: Performance depended on hyperparameters: ρ peaked at ρ = .01, values above the recommended range harmed coherence, λ affected topic births, and σ performed best from [5, 10].The sensitivity analysis varied one hyperparameter at a time while holding the others at default values; λ experiments fixed ∆ = 13.
6 Conclusions and Future work
The paper presents iDTM for time-varying document collections, adapting topic number, topic word distributions, and topic trends over time. It applies the model to NIPS proceedings to produce timelines of topic birth, evolution, trends, and representative papers.
- iDTM models time-varying document collections by adapting the number of topics, their word distributions, and their trends over time.
- The model was used to analyze NIPS conference proceedings and construct timelines of topic birth and evolution.
- For each topic, iDTM provides a timeline of its trend and the papers with the highest topic weight in their mixing vectors.
- These timelines provide a bird’s-eye view of the collection and can serve as input to a topic-level summarization system.
Appendix A: Fitting the proposal distribution in (14)
The appendix fits the proposal distribution for topic-word states using a Laplace approximation within forward and backward recurrences. The approximation restores a Gaussian form for continued inference despite non-conjugacy, while using a diagonal Hessian approximation for high-dimensional topic states.
- The target posterior is P(φ_1, . . . , φ_T | v_1, . . . , v_T), where v_t is the topic-word frequency count vector at epoch t.
- Because the model is a linear state-space model with nonlinear emission, the appendix uses a Laplace approximation rather than a closed-form RTS smoother solution.
- The forward recurrence approximates each state distribution as Gaussian, with the mode defining its mean and the negative inverse Hessian defining its covariance.
- A diagonal Hessian approximation is used because the topic-state vector is high-dimensional.
- The backward recurrence computes q(φ_t|v_1, . . . , v_T) and is exact when the dynamic model is linear, as in this model.