Source-linked AI summary
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!
Suzanna Sia, Ayush Dalmia, Sabrina J. Mielke
TL;DR
Topic modeling is the standard way to uncover themes in document collections, but the paper asks whether pretrained embeddings offer another option. It clusters word embeddings with document-informed weighting and reranking, benchmarks embedding–clustering combinations, and reports comparable performance to LDA with lower complexity and runtime.
Problem
Topic models are the standard approach for exploratory document analysis, motivating evaluation of whether pretrained embeddings provide an alternative.
Method
The paper clusters pretrained word-type embeddings, incorporates corpus frequency through weighted clustering and top-word reranking, and benchmarks embeddings, clustering algorithms, and PCA reduction.
Results
The best-performing combination is comparable with LDA, while using smaller time complexity and empirical runtime; PCA reduces embedding dimensions by up to 80% for further speedups.
Takeaways & Limitations
Pretrained contextualized and non-contextualized embeddings combined with tf-weighted k-means and tf-based reranking provide a viable lower-complexity alternative to traditional topic modeling.
Takeaways & Limitations
Training or fine-tuning embeddings on the given data could potentially improve performance, but this is left for future work.
Abstract
from arXiv · showhide
Topic models are a useful analysis tool to uncover the underlying themes within document collections. The dominant approach is to use probabilistic topic models that posit a generative story, but in this paper we propose an alternative way to obtain topics: clustering pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. We provide benchmarks for the combination of different word embeddings and clustering algorithms, and analyse their performance under dimensionality reduction with PCA. The best performing combination for our approach performs as well as classical topic models, but with lower runtime and computational complexity.
1 Introduction
The paper proposes clustering pretrained word embeddings as an alternative to probabilistic topic models, incorporating document statistics to improve topic quality. It benchmarks embeddings and clustering algorithms, finding comparable performance to LDA with lower complexity and runtime.
- Alternative approach: The approach represents word types with pretrained embedding vectors and clusters them by distance to obtain topics.It benchmarks modern embeddings and clustering methods against classical LDA rather than aiming strictly to outperform it.
- Alternative approach: Centroid-based algorithms provide top words by ranking cluster members according to distance from the cluster center.
- Method enhancements: Corpus frequency statistics are incorporated through term weighting during clustering and reranking terms for the top J representative words.These additions are intended to make the approach work reliably compared with LDA.
- Evaluation: The study systematically applies centroid-based clustering to varied pretrained word embeddings for unsupervised document analysis.
- Evaluation: The best-performing combination produces sensible topics comparable with LDA while using smaller time complexity and empirical runtime.
- Evaluation: PCA enables further speedups by reducing embedding dimensions by up to 80%.
2 Related Work and Background
Prior work has used embedding clustering for several document-analysis tasks, but its use for topic modeling has had limited success. This paper positions a systematic, centroid-focused comparison as a straightforward alternative for regular-length documents.
- Prior applications: Clustering word embeddings has supported readability assessment, argument mining, document classification, and document clustering.
- Algorithm choice: A cited drawback of hierarchical and density-based clustering is worse performance with more hyperparameters to tune than centroid-based alternatives.
- Topic modeling gap: Embedding clustering has seen limited success for topic modeling, while many studies instead incorporate embeddings into probabilistic LDA frameworks.
- Prior topic approaches: Earlier approaches cluster documents before identifying words, target short texts with GMMs, or use self-organising maps with only qualitative results.
- This paper’s scope: The paper focuses on centroid-based k-means, spherical k-means, k-medoids, von Mises–Fisher models, and Gaussian mixture models.It evaluates pretrained word2vec, GloVe, FastText, Spherical, and contextualized embeddings including ELMo.
3 Methodology
The paper obtains topics by clustering pretrained word embeddings, incorporating corpus statistics through weighting and reranking to produce more representative top words. It evaluates centroid-based clustering methods and focuses subsequent analysis on KM and GMM.
- Clustering pipeline: Each vocabulary word type is converted to an embedding, then clustered into k groups using weighted or unweighted types.Top J words are extracted from each cluster after convergence.
- Obtaining top words: Centroid-based methods select cluster topic words by proximity to a center or by probability under cluster parameters.KM and KD use distance, SK uses cosine similarity, and GMM/VMFM use likelihood.
- Algorithm selection: The main analysis focuses on KM and GMM because k-medoids, spherical KM, and von Mises-Fisher models perform worse.Those comparisons are reported in the appendices.
- Document-level topics: The method can also assign topics to a document by comparing its word embeddings with learned cluster centers and applying softmax normalization.The resulting distribution is explicitly described as non-calibrated.
- Corpus statistics: Weighting incorporates corpus frequency effects into clustering, while reranking addresses the possibility that nearest words are low-frequency types.The figure illustrates weighting as shifting the cluster center toward a more representative location; reranking then improves the selected top words.
- Corpus statistics: The experiments use tf weighting and reranking because tf outperforms the other tested weighting schemes.The same choice is used in the reported results and subsequent analysis.
4 Computational Complexity
The clustering approach has complexity based on iterations, topics, vocabulary size, and embedding dimensions, with weighting adding a constant-factor cost and reranking an additional sorting cost. Compared with LDA’s token-based complexity, clustering can offer a better performance-complexity tradeoff when the token count greatly exceeds the vocabulary size.
- Clustering complexity: KM has complexity O(tknm), while GMM has complexity O(tknm^3).Here t is iterations, k is clusters, n is unique vocabulary, and m is embedding dimensions.
- Weighting overhead: Weighted variants add a one-off initialization cost and a constant factor for recalculating centroids during clustering.The added cost does not change the stated asymptotic forms.
- Reranking overhead: Reranking adds O(n · log(nk)), where nk is the average number of elements in a cluster.This cost follows the clustering step.
- Comparison with LDA: LDA via collapsed Gibbs sampling has complexity O(tkN), where N is the number of all tokens.The comparison distinguishes vocabulary-based clustering from token-based LDA complexity.
- Comparison with LDA: When N ≫n, clustering methods can potentially achieve better performance-complexity tradeoffs than LDA.The analysis treats the maximum iteration count as a constant factor for worst-case analysis.
- Contextual embeddings: ELMo and BERT require a single document pass to produce contextual embeddings, rather than repeating token processing for every topic and clustering iteration.Generating these embeddings still requires passing through recurrent or transformer layers.
5 Experimental Setup
The evaluation uses 20NG and Reuters with unsupervised topic-word extraction on training data and NPMI evaluation on held-out data. It compares pretrained embeddings and clustering methods against LDA, with results averaged over five random seeds.
- Datasets: The 20NG dataset contains around 18,000 documents and 20 categories, while the Reuters subset contains around 10,000 documents.The implementation is freely available online.
- Data splits: The experiments use 60-40 train-test splitting for 20NG and 70-30 splitting for Reuters.Topic words are obtained from the training split and evaluated on the test split.
- Evaluation metric: NPMI evaluates the coherence of the top 10 topic words, ranges from −1 to 1, and is averaged across topics.A score of 1 indicates perfect association, and NPMI has been shown to correlate with human judgments.
- Topic configuration: Both datasets use 20 topics; this is optimal among 20, 50, and 100 topics for Reuters and matches the ground-truth count for 20NG.The topic words are selected without using document labels.
- Reported results: 0.15 is the reported NPMI score for BERT on Reuters, while 0.26 is achieved by both LDA and KMw Spherical on 20NG.The results are averaged across five random seeds.
- Preprocessing: Preprocessing lowercases tokens, removes stopwords, punctuation, and digits, and excludes rare words and long noisy sentences.Words appearing in fewer than 5 documents or in sentences longer than 50 words are excluded.
- Contextual embeddings: For BERT and ELMo, sentences provide context windows, and BERT subword representations are averaged.Averaging subwords performs better than using only the first subword.
6 Results and Discussion
Reranking and term-frequency weighting substantially improve embedding-clustering topics, while the strongest combinations match LDA coherence with lower runtime and reduced dimensionality. Performance varies by embedding and clustering algorithm, with GMM often stronger than KM but KM preferable for practical efficiency.
- LDA takes about a minute, whereas KMwr takes little more than 10 seconds on CPU and 3–4 seconds with a simple JAX implementation on GPU, with no better LDA performance.
- GMM outperforms KM for weighted and unweighted variants averaged across all embedding methods (p < 0.05).
- BERT-GMMw has an average top-word Jaccard similarity of 0.910 before and after reranking, indicating weighted GMM centers already favor frequent training-corpus words.
- 0.15 NPMI is achieved by BERT-GMMw on Reuters, compared with 0.12 for LDA; on 20NG, Spherical-KMr and LDA both achieve 0.26 NPMI.
- The approach yields greater topic diversity than LDA while achieving comparable coherence scores.
- More than 80% of embedding dimensions can be removed with reranking, while KMwr is favored over GMMs for efficiency because GMMwr has cubic complexity.
7 Conclusion
The paper presents clustering pretrained word embeddings as a systematic alternative to traditional topic modeling. Its experiments support tf-weighted k-means with tf-based reranking as viable at lower complexity and runtime.
- Pretrained embeddings combined with tf-weighted k-means and tf-based reranking provide a viable alternative to traditional topic modeling at lower complexity and runtime.
A k-means (KM) vs k-medoids (KD)
K-medoids is evaluated as a less outlier-sensitive alternative to k-means, but generally does not improve topic coherence. Its occasional advantages remain small and below the strongest models.
- K-medoids is a hard clustering algorithm similar to k-means but less sensitive to outliers.
- K-medoids usually performs as well as or worse than k-means and does relatively poorly after frequency reranking.
- Where k-medoids outperforms k-means, the difference is not striking and NPMI remains below the other top-performing models.
B Results for Spherical k-means and Von Mises-Fisher Mixture
The appendix compares alternative spherical clustering methods and reranking schemes. Spherical k-means and von Mises–Fisher mixtures perform poorly overall, while TF reranking remains stronger than alternative reranking schemes.
- Spherical k-means and von Mises–Fisher mixtures show overall poor performance.
- Aggregated TF-IDF and TF-DF reranking improve over original hard clustering but perform worse than TF reranking.
D Qualitative Comparison of Topics Generated
The qualitative comparison contrasts LDA topics with clustered embedding-based topics, including BERT KMwᵣ on Reuters. Clustering assigns each word to one topic and yields greater diversity across topics.
- D Qualitative Comparison of Topics Generated: Hard clustering assigns each word to one topic, unlike LDA, which can place duplicate words in duplicate topics.This assignment difference affects how words are distributed across generated topics.
- D Qualitative Comparison of Topics Generated: BERT KMwᵣ introduces location- and corporate-position topics beyond LDA topics that mainly concern wealth and profits.The passage describes this as a qualitative difference between the generated topic sets.
- D Qualitative Comparison of Topics Generated: Clustering discovers greater topic diversity because its words are more diverse across topics.The paper links this diversity to the distribution of words over the topic set.
- D Qualitative Comparison of Topics Generated: The approach achieves comparable coherence scores to LDA while producing greater within-topic diversity.The paper states that such topics may be more valuable for exploratory analysis.
- D Qualitative Comparison of Topics Generated: LDA and BERT KMwᵣ topics are presented for qualitative comparison on the Reuters dataset.The comparison includes NPMI scores and each topic’s top 10 words.