PaperlayerView exact revision on arXiv

Source-linked AI summary

A correlated topic model of Science

David M. Blei, John D. Lafferty

arXiv:0708.3601v2stat.AP

TL;DR

LDA does not directly model correlations among topics, although such relationships are common in document collections. The paper introduces the CTM, using a logistic normal and fast variational inference, and finds better fit than LDA while enabling exploratory analysis of a large Science archive.

  • Problem

    LDA cannot directly model correlations among topics because its Dirichlet topic-proportion distribution imposes near independence.

  • Method

    The CTM replaces the Dirichlet with a logistic normal and uses fast variational inference for approximate posterior inference.

  • Results

    The CTM gives a better fit than LDA on JSTOR Science data, measured by predictive distributions over held-out documents.

  • Takeaways & Limitations

    The CTM captures correlated latent topics and supports exploratory browsing, search, topic graphs, and analysis of large document collections.

  • Takeaways & Limitations

    The paper leaves topic-number selection insufficiently explored because the logistic normal does not immediately support Dirichlet-process extensions.

Abstract

from arXiv · show

Topic models, such as latent Dirichlet allocation (LDA), can be useful tools for the statistical analysis of document collections and other discrete data. The LDA model assumes that the words of each document arise from a mixture of topics, each of which is a distribution over the vocabulary. A limitation of LDA is the inability to model topic correlation even though, for example, a document about genetics is more likely to also be about disease than X-ray astronomy. This limitation stems from the use of the Dirichlet distribution to model the variability among the topic proportions. In this paper we develop the correlated topic model (CTM), where the topic proportions exhibit correlation via the logistic normal distribution [J. Roy. Statist. Soc. Ser. B 44 (1982) 139--177]. We derive a fast variational inference algorithm for approximate posterior inference in this model, which is complicated by the fact that the logistic normal is not conjugate to the multinomial. We apply the CTM to the articles from Science published from 1990--1999, a data set that comprises 57M words. The CTM gives a better fit of the data than LDA, and we demonstrate its use as an exploratory tool of large document collections.

1. Introduction.

The paper introduces the correlated topic model (CTM) to address LDA’s inability to represent correlations among latent topics. It develops fast variational inference and applies the CTM to large Science archives for predictive evaluation and exploratory analysis.

  • Motivation: Topic models extract interpretable semantic structure from large document collections without explicitly understanding language.The motivation includes increasingly accessible scholarly archives and the need for automated browsing, searching, and statistical analysis.
  • Background: LDA models documents as mixtures of topics but assumes nearly independent topic proportions through the Dirichlet distribution.Each topic is a multinomial over a fixed vocabulary, while topic proportions vary stochastically across documents.
  • Contribution: The CTM replaces the Dirichlet with a logistic normal so the covariance structure can represent correlations among latent topics.For example, genetics may correlate with health and disease more than with X-ray astronomy.
  • Inference: The paper develops a fast variational inference procedure because replacing the Dirichlet removes conjugacy and complicates posterior inference.Variational inference trades potentially unbiased MCMC estimates for computational efficiency and assessable numerical convergence.
  • Evaluation and application: The CTM gives a better fit than LDA on JSTOR Science data and supports topic-related article analysis and covariance-based topic graphs.The application covers Science articles published from 1990–1999 and demonstrates exploratory analysis of the collection.

2. The correlated topic model.

The CTM generates document topic proportions by transforming a multivariate Gaussian through the logistic normal distribution. This covariance structure captures dependencies among topics, providing a more expressive alternative to LDA’s Dirichlet proportions at the cost of more difficult inference.

  • Model structure: The CTM is a hierarchical mixed-membership model in which shared topics combine with document-specific proportions.Documents can exhibit multiple topics with different proportions, capturing heterogeneity in grouped data.
  • Generative process: The CTM’s generative process draws topic assignments from document proportions and words from the multinomial distribution of each assigned topic.Topics are distributions over the vocabulary, and each document’s proportions determine probabilities over topic indices.
  • Logistic normal: The logistic normal draws a real-valued vector from a multivariate Gaussian and maps it to the simplex to obtain multinomial topic proportions.The Gaussian covariance induces dependencies among the transformed proportions.
  • Correlation: Unlike the Dirichlet, the logistic normal captures general covariance patterns among topic proportions.This allows relationships such as geology being associated with archeology more than genetics.
  • Trade-off: Modeling correlation complicates inference because the logistic normal is not conjugate to the multinomial.The paper accepts this computational drawback to obtain a more expressive document model and use covariance for corpus exploration.

3. Computation with the correlated topic model.

The CTM requires approximate posterior inference and parameter estimation because its logistic-normal topic proportions are nonconjugate to the multinomial. The paper develops variational procedures and uses covariance structure to represent relationships among topics.

  • Posterior inference: The CTM estimates document-level latent variables with a fast variational inference algorithm, embedding new documents into the model’s latent thematic space.The posterior is conditioned on document words, topics, and logistic-normal parameters.
  • Posterior inference: The posterior is intractable because topic assignments create combinatorial terms and the logistic normal is nonconjugate to topic assignments.Consequently, the required integrals cannot be computed analytically.
  • Posterior inference: Variational inference uses a fully factorized distribution whose topic assignments have multinomial parameters and whose continuous variables have independent univariate Gaussian factors.The algorithm optimizes a likelihood bound, equivalently minimizing KL(q||p) relative to the true posterior.
  • Parameter estimation: Variational EM estimates the topics and multivariate Gaussian parameters by alternating document-level variational inference with model-parameter maximization.The E-step updates variational parameters, while the M-step uses expected sufficient statistics.
  • Topic graphs: The CTM’s covariance structure supports topic graphs, while increasing the regularization parameter ρ_n makes those graphs increasingly sparse.The graph represents topics as nodes and connects neighboring topics identified from covariance-based estimates.

4. Analyzing Science.

The CTM is applied to Science articles as an exploratory model of topic structure, document similarity, and predictive performance. Its inferred topic correlations support topic graphs and improve held-out fit and prediction relative to LDA.

  • Corpus and model: The analysis estimates a 100-topic CTM on 16,351 Science articles from 1990–1999 after vocabulary pruning.The collection contains 375,144 terms before removing rare terms and stop words.
  • Qualitative analysis: A sparse topic graph reveals substructures of themes and lets users focus on neuroscience or genetics topic clusters.The graph was constructed with ρ = 0.1; increasing the regularization parameter makes the graph sparser.
  • Document exploration: Documents can be associated with latent topics and ranked for similar topical content using expected Hellinger distance between topic-proportion vectors.Figure 3 illustrates the top three articles similar to a query about Earth’s solid iron core, with each document combining geology and physics.
  • Quantitative comparison: The CTM fits the 1960 Science corpus better than LDA, with likelihood peaking near 90 topics for CTM versus near 30 for LDA.Held-out log probability was estimated with ten-fold cross-validation and importance sampling.
  • Quantitative comparison: Predictive perplexity improves by nearly 200 words, or roughly 10%, for the CTM when only a small number of document words are observed.Lower perplexity indicates greater predictive power; the CTM uses correlations to infer probable words from related topics.

5. Summary.

The paper replaces Dirichlet topic proportions with a logistic normal to model correlations among latent topics. This model supports descriptive statistics for browsing and search, while model selection remains insufficiently explored.

  • The correlated topic model replaces the Dirichlet distribution of per-document topic proportions with a logistic normal.This allows correlations between the occurrence of latent topics to be represented.
  • The logistic normal enables the model to capture correlations between latent topics and may benefit Dirichlet-based mixed membership applications.The paper characterizes this increased flexibility as more complex.
  • Model selection, including choosing the number of topics, was not thoroughly explored and remains an important area for future research.The logistic normal does not immediately support nonparametric Bayesian extensions analogous to those available for LDA.

APPENDIX: DETAILS OF VARIATIONAL INFERENCE

The variational inference procedure optimizes a lower bound using an additional variational parameter and coordinate ascent. Nonconjugacy requires bounding the negative log normalizer, and joint optimization is not guaranteed to be globally convex.

  • Variational objective: The variational objective is expressed in terms of variational parameters for the topic proportions, assignments, and auxiliary parameter ζ.The method optimizes λ1:K, ν1:K, φ1:N, and ζ.
  • Variational objective: The logistic normal’s nonconjugacy with the multinomial complicates the expected log probability of topic assignments.The procedure preserves a lower bound by upper-bounding the negative log normalizer with a Taylor expansion.
  • Variational objective: The Taylor-bound construction introduces ζ and uses Eq[exp{ηi}], the mean of a log normal distribution determined by λi and νi^2.The resulting approach is simpler than a more flexible but more computationally intensive alternative.
  • Coordinate ascent optimization: Coordinate ascent iteratively maximizes the bound with respect to ζ, φ, λ, and ν.Updates for φ use variational inference within the exponential family; λ uses conjugate gradients, while ν uses Newton’s method with νi > 0.
  • Coordinate ascent optimization: Each coordinate optimization is convex, but the joint variational objective is not, so the algorithm is guaranteed only to find a local maximum.That local maximum still provides a bound on a document’s log probability.
Loading 0708.3601v2…