Source-linked AI summary
Topic Modeling in Embedding Spaces
Adji B. Dieng, Francisco J. R. Ruiz, David M. Blei
TL;DR
Large vocabularies make existing topic models difficult to interpret and can require pruning important terms. The ETM combines topic modeling with word embeddings and uses efficient inference, achieving interpretable topics and strong predictive performance across large-vocabulary settings.
Problem
LDA fails with large vocabularies, forcing pruning that may remove important terms and limit model scope.
Method
The ETM models documents generatively with topics and words in a shared embedding space, using categorical word distributions and amortized variational inference.
Results
The ETM learns interpretable topics and accurate word distributions, including in corpora with large vocabularies and stop words.
Takeaways & Limitations
The ETM combines interpretable topic structure with low-dimensional word representations while robustly accommodating large vocabularies and long-tail language data.
Takeaways & Limitations
The model replaces the Dirichlet with a logistic normal to facilitate reparameterization during inference.
Abstract
from arXiv · showhide
Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the Embedded Topic Model (ETM), a generative model of documents that marries traditional topic models with word embeddings. In particular, it models each word with a categorical distribution whose natural parameter is the inner product between a word embedding and an embedding of its assigned topic. To fit the ETM, we develop an efficient amortized variational inference algorithm. The ETM discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation (LDA), in terms of both topic quality and predictive performance.
1 Introduction
Topic modeling seeks latent semantic structure, but LDA degrades with large vocabularies. The ETM combines topic modeling with word embeddings to retain interpretability and predictive performance in this setting.
- Large vocabularies expose a technical weakness in LDA, requiring pruning that can remove important terms and limit model scope.
- The ETM combines topic modeling with word embeddings, representing topics as points and terms as embeddings in a shared space.
- The ETM provides interpretable latent semantic structure, low-dimensional word representations, and robustness to large vocabularies and long-tail language data.
- As vocabulary size increases, LDA’s predictive performance and topic quality deteriorate, whereas the ETM maintains good performance.Figure 1 uses normalized held-out perplexity and topic coherence on 20NewsGroup with 100 topics.
- The ETM is robust to stop words and produces topics that remain interpretable when they are included in the vocabulary.
- The ETM uses variational and amortized inference to fit large corpora with large vocabularies.Its embeddings may be pretrained or learned jointly with the remaining parameters.
2 Related Work
Related approaches extend LDA with word similarity, embeddings, or neural inference. The ETM instead models categorical text through an embeddings matrix and can learn embeddings during inference.
- Prior methods incorporate word similarity by modifying topic priors or topic-assignment priors.
- Embedding-based extensions commonly convert discrete text into continuous observations and model topics with Gaussian or von Mises–Fisher likelihoods.
- The ETM models categorical data through the embeddings matrix, rather than generating continuous embedding observations.
- Unlike approaches requiring pretrained embeddings, the ETM can learn embeddings jointly with its inference process.
- Other work combines LDA likelihoods with pretrained embeddings, replaces topic words with Gaussian embeddings, or jointly learns topics and embeddings geometrically.
3 Background
The background presents LDA as a probabilistic topic model and word embeddings as distributed representations of meaning. The ETM builds on both ideas.
- Latent Dirichlet allocation: LDA represents each topic as a vocabulary distribution and each document as a mixture of shared topics.
- Latent Dirichlet allocation: LDA generates document topic proportions from a Dirichlet distribution, assigns each word a topic, and draws the word from that topic’s categorical distribution.
- Latent Dirichlet allocation: LDA places Dirichlet priors on topics, with concentration parameters treated as fixed hyperparameters.
- Word embeddings: Word embeddings represent vocabulary terms as vectors whose proximity reflects similarities in meaning derived from usage contexts.
- Word embeddings: CBOW models each word using an embedding matrix and a context embedding formed by summing embeddings of surrounding words.
4 The Embedded Topic Model
The ETM represents words and topics in a shared embedding framework while retaining document-level topic mixtures. Its log-linear word distribution uses embedding agreement, with logistic-normal proportions and optional pretrained embeddings.
- Model representation: The ETM embeds vocabulary terms in an L-dimensional space and represents each topic as a vector in that same space.Each topic embedding is a distributed representation of a topic in the words’ semantic space.
- Topic distributions: Unlike traditional topic models, ETM induces each topic’s vocabulary distribution from word embeddings and a topic embedding rather than storing a full distribution directly.The induced distribution is computed with softmax(ρ⊤αk).
- Word generation: ETM word probabilities follow a log-linear form based on inner products between word embeddings and assigned topic embeddings.Higher embedding agreement produces higher probability for a word under the topic.
- Generative process: The model draws document topic proportions, assigns each word a topic, then generates the word from that topic.Topic proportions use a logistic-normal distribution, replacing the Dirichlet to support reparameterization.
- Embedding choices: The ETM can learn word embeddings jointly with topics or use pretrained embeddings to model words absent from the corpus.With pretrained embeddings, it can calculate topic-word compatibility even for unused vocabulary terms.
5 Inference and Estimation
ETM estimation maximizes document likelihood using variational inference because the marginal likelihood is intractable. Amortized inference networks, stochastic optimization, and minibatches make fitting scalable to document collections.
- Objective: The ETM parameters are word embeddings and topic embeddings, which are fitted by maximizing the marginal likelihood of the documents.Each topic embedding is a point in the embedding space.
- Inference challenge: The document marginal likelihood is intractable because it requires integrating over topic proportions.The method therefore replaces direct likelihood computation with a variational lower bound.
- Amortized inference: Variational inference optimizes per-document bounds using document-dependent Gaussian distributions produced by a shared neural inference network.The network outputs the mean and variance of the untransformed topic proportions.
- ELBO: The ELBO balances explaining observed words against keeping variational topic proportions close to their prior.Both model and variational parameters are optimized.
- Optimization: Stochastic optimization uses Monte Carlo reparameterization gradients, data subsampling, and Adam to fit large document collections.Algorithm 1 applies minibatches and updates both model and variational parameters.
6 Empirical Study
The empirical study evaluates ETM against LDA and NVDM on predictive performance, topic quality, vocabulary scaling, and stop-word robustness. ETM provides interpretable topics while achieving strong predictive performance, including with large vocabularies and stop words.
- Evaluation setup: The study compares ETM with LDA and NVDM using predictive accuracy and topic interpretability across two news corpora.Predictive accuracy uses document-completion log-likelihood, while topic quality combines coherence and diversity.
- Evaluation setup: The experiments use 20Newsgroups and New York Times corpora with vocabulary sizes ranging from 3,102 to 52,258 and from 5,921 to 212,237, respectively.The New York Times corpus contains more than 1.8 million articles spanning 1987–2007.
- Qualitative results: ETM and LDA provide interpretable topics, whereas NVDM and Δ-NVDM do not provide interpretable topics under the reported topic inspection.The ETM and NVDM learn embeddings similar to skip-gram embeddings, while the simplex constraint changes Δ-NVDM embeddings.
- Quantitative results: Among interpretable models, ETM provides the best predictions while retaining interpretable topics and robustness to large vocabularies.Both ETM variants are at least as good in prediction as the simplex-constrained Δ-NVDM.
- 6.1 Stop words: Standard topic models place stop words in almost every topic, while labeled ETM isolates them using their location in the embedding space.The labeled ETM achieves the best topic quality among the evaluated stop-word models with K = 300 topics.
7 Conclusion
The ETM combines topic modeling with word embeddings by generating words from the inner product of word and assigned-topic embeddings. It learns interpretable topics and word embeddings in large-vocabulary corpora while maintaining topic quality with stop words.
- The ETM is a generative document model that combines LDA with word embeddings.Topics and words occupy the same embedding space.
- The labeled ETM remains robust to stop words, achieving similar topic coherence to the setting without stop words.Topic quality is defined as the product of topic coherence and topic diversity, with higher values better.
- Words are generated from a categorical distribution whose natural parameter is the inner product of word and assigned-topic embeddings.
- The ETM learns interpretable word embeddings and topics even with large vocabularies.