Source-linked AI summary
How Many Topics? Stability Analysis for Topic Models
Derek Greene, Derek O'Callaghan, Pádraig Cunningham
TL;DR
Topic modeling needs a suitable number of topics, but existing choices can be overly broad or over-clustered. The paper proposes term-centric stability analysis using ranked top terms across perturbed matrix-factorization runs, and evaluations suggest it can guide one or more suitable k values. The method is demonstrated with NMF and may generalize to approaches that represent topics as ranked term lists.
Problem
Selecting an appropriate number of topics k is difficult because too few topics are overly broad and too many produce over-clustering.
Method
The method measures agreement among top-term rankings generated across document samples and random matrix initializations, using a top-weighted ranking measure.
Results
Evaluations on a range of corpora suggest that term-centric stability can guide selection of one or more values for k.
Takeaways & Limitations
Because it compares ranked terms rather than raw factor values or probabilities, the approach can potentially generalize beyond NMF to LDA and document-clustering algorithms.
Takeaways & Limitations
The experiments focus on NMF, with alternative algorithms left for further study.
Abstract
from arXiv · showhide
Topic modeling refers to the task of discovering the underlying thematic structure in a text corpus, where the output is commonly presented as a report of the top terms appearing in each topic. Despite the diversity of topic modeling algorithms that have been proposed, a common challenge in successfully applying these techniques is the selection of an appropriate number of topics for a given corpus. Choosing too few topics will produce results that are overly broad, while choosing too many will result in the "over-clustering" of a corpus into many small, highly-similar topics. In this paper, we propose a term-centric stability analysis strategy to address this issue, the idea being that a model with an appropriate number of topics will be more robust to perturbations in the data. Using a topic modeling approach based on matrix factorization, evaluations performed on a range of corpora show that this strategy can successfully guide the model selection process.
1 Introduction
Topic modeling requires choosing a suitable number of topics because too few topics are overly broad, too many over-cluster the corpus, and some corpora support multiple resolutions. The paper proposes selecting k through term-ranking stability under data and initialization perturbations.
- Selecting k is central to topic modeling: low values produce overly broad topics, while high values produce over-clustering.
- Some corpora contain coherent topics at multiple resolutions, allowing several appropriate values of k.
- Stability measures whether repeated analyses on perturbed data consistently reproduce similar solutions.
- The proposed method compares top-term rankings across runs using document sampling and random matrix initialization, with greater weight on higher-ranked terms.
- The approach is presented with matrix-factorization topic modeling and evaluated empirically on a range of text corpora.
2 Related Work
Related work covers NMF and stability-based model selection, alongside ranking-similarity and topic-model evaluation measures. These approaches generally select model parameters by assessing reconstruction, repeated-solution agreement, consensus structure, or top-term coherence.
- Matrix Factorization: NMF decomposes a non-negative document-term matrix into non-negative factors, with k controlling the reduced-rank approximation.
- Matrix Factorization: NMF topics support overlapping document memberships and can use TF-IDF-weighted, length-normalized matrices rather than only raw frequency counts.
- Matrix Factorization: NMF model selection lacks a definitive method; RSS curves offer a heuristic based on identifying an inflection point across candidate k values.
- Stability Analysis: Stability analysis perturbs data and selects k values with the highest mean agreement across repeated clusterings.
- Stability Analysis: Brunet et al. measure NMF stability through consensus connectivity matrices and cophenetic correlations, recommending k near a sudden correlation drop.
- Topic-Model Evaluation: LDA stability work compares topics across runs using distances between topic distributions, while coherence evaluations assess top-term semantic relatedness.
- Rank Agreement: Ranked-list comparison methods include Spearman’s footrule, Kendall’s tau, and top-weighted measures designed for indefinite rankings.
3 Methods
The method evaluates topic-model stability through agreement between ranked term lists, then selects topic counts by examining stability across document-resampled models. It uses Average Jaccard for top-weighted term similarity and optimal matching to compare topic models.
- Term Ranking Similarity: The approach represents each topic model as a set of k ranked lists containing its top t characteristic terms.For NMF, rankings come from basis-vector values; for LDA, they come from topic-term probabilities.
- Term Ranking Similarity: Average Jaccard compares indefinite ranked term lists by averaging Jaccard scores across prefix depths, giving greater weight to highly ranked terms.The measure is symmetric and ranges from 0 to 1.
- Topic Model Agreement: Topic-model agreement is computed by building a k × k Average Jaccard similarity matrix and finding the optimal topic matching with the Hungarian method.The resulting agreement score ranges from 0 to 1, with identical models scoring 1.
- Selecting the Number of Topics: For each candidate k, the method compares a reference model trained on the full corpus with τ models trained on random samples of β × n documents.Each model is represented by ranked lists of its top t terms, and stability is the mean agreement across the reference-to-sample comparisons.
- Selecting the Number of Topics: Candidate topic counts are identified from peaks in the resulting stability plot, while multiple peaks indicate multiple appropriate topic schemes.A flat curve with low stability suggests that the corpus lacks coherent topics.
4 Evaluation
The evaluation applies stability analysis and consensus-matrix selection across annotated text corpora, comparing candidate topic counts from k = 2 to 12. Results show that stability often identifies meaningful granularities, while also revealing coarse merged-topic solutions, ground-truth mismatches, and computational advantages over consensus analysis.
- Experimental setup: The evaluation compares stability analysis with consensus-matrix analysis across eight annotated corpora and k ∈[2, 12].Both methods use the same collection of NMF factorizations, enabling a direct comparison of their selection plots.
- Top-term sensitivity: t = 20 is used thereafter because scores for t = 10/20/50/100 were highly correlated, with a highest pairwise mean correlation of 0.977.The plots nearly overlap because lower-ranked terms contribute less to the Average Jaccard agreement measure; t = 20 also provides economy of representation.
- Corpus results: k = 5 is a strong peak for both methods on bbc, matching its five annotated categories, whereas bbcsport peaks at k = 4 before a lower peak at k = 5.For bbcsport, the ranking evidence indicates that athletics and tennis are merged while the other three categories remain distinct.
- Corpus results: The stability method identifies k = 6 for guardian-2013, matching its six sections, while both methods also indicate a coarser k = 3 organization.At k = 3, culture-related sections and politics/business are merged, while football remains distinct.
- Corpus results: For irishtimes-2013, stability identifies k = 7 alongside a high k = 2 peak; nytimes-1999 instead has a trough at k = 4 despite four ground-truth categories.The nytimes-1999 instability is associated with sports splitting inconsistently into baseball and basketball topics across factorization results.
- Discussion: Both methods frequently peak at k = 2 because repeated outlier and merged topics produce high agreement, while metadata-based ground truth may not match the data's actual structure.The consensus scores exceeded 0.8 and were min-max normalized for plotting; consensus analysis also required substantially longer runtimes than the reference-ranking stability method.
5 Conclusion
The paper proposes term-centric stability analysis to guide selection of the number of topics, using agreement among top-ranked terms across corpus samples. Evaluations suggest the method can identify one or more suitable values of k, while its experiments focused on NMF and its broader applicability remains potential.
- Term-centric stability analysis selects k by measuring agreement among top-ranked topic terms generated from different samples of the same corpus.
- Evaluations on a range of text corpora suggest that the method can provide a useful guide for selecting one or more values of k.
- The experiments focused on NMF, although the ranking-based approach could potentially generalize to topic models that represent topics as ranked term lists.