Source-linked AI summary
BERTopic: Neural topic modeling with a class-based TF-IDF procedure
Maarten Grootendorst
TL;DR
Existing topic models can miss semantic relationships through bag-of-words representations, motivating contextual approaches. BERTopic embeds and clusters documents, then uses class-based TF-IDF to generate topic representations; it learns coherent language patterns and performs competitively and stably across varied tasks. The authors also identify limitations involving single-topic document assumptions and bag-of-words topic representations.
Problem
Bag-of-words topic models disregard semantic relationships and word context, limiting their representation of documents.
Method
BERTopic creates pretrained-language-model document embeddings, clusters reduced embeddings with HDBSCAN, and extracts topic representations using class-based TF-IDF.
Results
BERTopic learns coherent patterns of language and demonstrates competitive and stable performance across a variety of tasks.
Takeaways & Limitations
Separating document clustering from topic-representation generation gives BERTopic significant flexibility and ease of usability.
Takeaways & Limitations
BERTopic assumes each document contains a single topic, and its topic representations remain based on bag-of-words rather than directly accounting for contextual representations.
Abstract
from arXiv · showhide
Topic models can be useful tools to discover latent topics in collections of documents. Recent studies have shown the feasibility of approach topic modeling as a clustering task. We present BERTopic, a topic model that extends this process by extracting coherent topic representation through the development of a class-based variation of TF-IDF. More specifically, BERTopic generates document embedding with pre-trained transformer-based language models, clusters these embeddings, and finally, generates topic representations with the class-based TF-IDF procedure. BERTopic generates coherent topics and remains competitive across a variety of benchmarks involving classical models and those that follow the more recent clustering approach of topic modeling.
1 Introduction
Conventional topic models use bag-of-words representations that overlook semantic relationships and word context. BERTopic addresses this limitation by combining contextual document embeddings, clustering, and class-based TF-IDF topic representations.
- Bag-of-words representations disregard semantic relationships and word context, which can make documents inaccurately represented.
- Transformer-based models such as BERT generate contextual word and sentence representations that encode semantic properties.
- Embedding-based topic modeling clusters semantically represented documents but may derive topic words from proximity to a cluster centroid.
- Centroid-based topic representations can mislead when document clusters do not have spherical geometry, even after frequency-based reranking.
- BERTopic uses pretrained language-model embeddings, dimensionality reduction, document clustering, and class-based TF-IDF to generate topic representations.Its independent stages support flexible use cases, including dynamic topic modeling.
2 Related Work
Recent topic-modeling approaches cluster word or document embeddings, separating cluster formation from topic representation. BERTopic extends this approach with a class-based TF-IDF procedure for creating topic representations.
- Neural topic models increasingly use neural networks and word embeddings to improve topic modeling beyond classical approaches.
- Clustering word and document embeddings separates cluster generation from topic-representation generation, making topic modeling more flexible.
- BERTopic extends embedding-based clustering by incorporating a class-based variant of TF-IDF for topic representations.
3 BERTopic
BERTopic embeds documents, reduces embedding dimensionality, clusters documents, and extracts topic representations with class-based TF-IDF. The procedure treats each cluster as a class so word importance is measured at the topic level.
- BERTopic: BERTopic converts documents into pretrained-language-model embeddings, reduces their dimensionality, clusters them, and extracts topic representations with class-based TF-IDF.
- Document clustering: UMAP reduces document-embedding dimensionality before clustering, while HDBSCAN finds clusters with varying densities and models noise as outliers.
- Topic Representation: The topic-representation procedure modifies TF-IDF so term importance is measured for a topic rather than an individual document.
- Topic Representation: Documents in each cluster are concatenated into a single class, and inverse document frequency is replaced by inverse class frequency.
- Topic Representation: Class-based TF-IDF models word importance across clusters and produces topic-word distributions for each cluster.
- Topic Representation: BERTopic can iteratively merge the least common topic with its most similar topic to reach a user-specified number of topics.
4 Dynamic Topic Modeling
BERTopic models temporal topic representations by fitting global topics first and deriving local representations at each timestep. These local representations can be computed without re-embedding or reclustering, while optional smoothing incorporates adjacent timesteps.
- Dynamic Topic Modeling: Traditional topic models are static, whereas dynamic topic models represent how topics evolve over time.
- Dynamic Topic Modeling: BERTopic first fits the entire corpus to create global topics, then derives local topic representations for documents at each timestep.
- Dynamic Topic Modeling: Local temporal representations can be computed without embedding and clustering documents again, enabling fast computation and use with metadata such as authors or journals.
- Smoothing: Because each timestep’s representation is independent of the previous one, the resulting dynamic topics may not evolve linearly.
- Smoothing: Optional smoothing averages normalized c-TF-IDF vectors at adjacent timesteps to incorporate the preceding representation.
5 Experimental Setup
BERTopic was evaluated on three datasets with different preprocessing choices and compared with several topic-modeling baselines. The implementation and experimental setup were made freely available.
- Reproducibility: The implementation of BERTopic and the experimental setup were made freely available online.
- Datasets: BERTopic was evaluated on 20 NewsGroups, BBC News, and Trump’s tweets.The datasets were selected to cover news articles and more recent short-text data.
- Datasets: 20 NewsGroups contains 16309 articles across 20 categories, while BBC News contains 2225 documents from 2004–2005.Both datasets were retrieved using OCTIS and underwent punctuation removal, lemmatization, stopword removal, and filtering of documents shorter than five words.
- Datasets: Trump’s dataset contains 44253 non-retweet tweets collected between 2009 and 2021.The tweets were selected for dynamic topic modeling because they have an inherent temporal structure.
- Baselines: BERTopic was compared with LDA, NMF, CTM, and Top2Vec using Doc2Vec and SBERT-based variants.LDA and NMF were run through OCTIS with default parameters, while the all-mpnet-base-v2 SBERT model was used for BERTopic and CTM.
20 NewsGroups BBC News Trump
The experiments evaluate topic models across datasets, language models, dynamic settings, and computational cost using coherence and diversity measures. These measures provide useful indications but remain imperfect proxies for subjective human evaluation.
- Language Models: BERTopic was tested with Universal Sentence Encoder, Doc2Vec, all-MiniLM-L6-v2, and all-mpnet-base-v2 language models.This setup evaluates generalizability across different embedding models.
- Dynamic Topic Modeling: Dynamic-topic experiments compared BERTopic, with and without linearly evolving topics, against LDA Sequence.
- Evaluation Metrics: Topic coherence was measured with NPMI, while topic diversity was measured as the percentage of unique words across topics.NPMI ranges from -1 to 1, and topic diversity ranges from 0 to 1.
- Evaluation Protocol: NPMI was evaluated from 10 to 50 topics in steps of 10, with results averaged across 3 runs for each step.Dynamic topic models were evaluated at 50 topics for each timestep and then averaged.
- Evaluation Limitations: Topic coherence and diversity are only indicative because they proxy subjective judgments that can differ between users.The paper also notes that NPMI’s relationship with human judgment may not hold for neural topic models.
- Computational Cost: Wall-time experiments used a two-core Intel Xeon CPU and a Tesla P100 GPU to assess practical running-time differences.Wall time was chosen because it more closely reflects practical use than narrower computational measures.
6 Results
BERTopic generally achieved high coherence and competitive diversity across datasets, remained stable across embedding models and dynamic settings, and offered a speed–performance trade-off with MiniLM. Its wall-time results were measured in a GPU-enabled environment.
- 6.1 Performance: BERTopic generally achieved high topic coherence across datasets and was especially strong on Trump’s tweets.It remained competitive on the more thoroughly preprocessed 20 NewsGroups and BBC News datasets, while CTM consistently achieved higher topic diversity.
- 6.2 Language Models: BERTopic’s coherence and diversity remained stable across the four tested SBERT language-model configurations.The results suggest that all-MiniLM-L6-v2 may be preferable when GPU capacity is limited because it is smaller and faster.
- 6.2 Language Models: Top2Vec’s performance changed substantially with its embedding choice: Doc2Vec was competitive for coherence, whereas MPNET reduced both coherence and diversity.The paper attributes this difference to Doc2Vec jointly embedding words and documents in the same space.
- 6.2 Language Models: BERTopic remained competitive across embedding models by separating document embedding from construction of the word-topic distribution.
- 6.3 Dynamic Topic Modeling: BERTopic performed consistently well in dynamic topic modeling, outperforming LDA on all measures for Trump and achieving the highest coherence for the UN dataset.
- 6.3 Dynamic Topic Modeling: The linearly evolving-topic assumption did not affect topic coherence or diversity on either dynamic dataset.
- 6.4 Wall time: NMF and LDA were faster than neural topic models, while MiniLM offered a speed–performance trade-off comparable in speed to Doc2Vec.BERTopic and Top2Vec had similar wall times when using the same language models.
- 6.4 Wall time: Wall-time conclusions were obtained with a GPU available for document embedding, and times are expected to increase significantly without one.Doc2Vec is an alternative, but its stability for coherence and diversity was questioned in the experiments.
7 Discussion
BERTopic remains competitive and flexible across language models, preprocessing choices, and topic-generation settings, while its topic assumptions impose important limitations.
- Scope: The study does not cover all evaluation metrics or use cases relevant to topic modeling.The authors discuss strengths and weaknesses to clarify when BERTopic may or may not be appropriate.
- Strengths: Separating document embedding from topic representation enables independent preprocessing and topic-generation fine-tuning.Topic representations can use different stopword settings or larger n-grams without re-clustering.
- Strengths: Class-based TF-IDF represents topics as word distributions, supporting modeling of topic evolution and representations across classes.These distributions allow dynamic and class-specific topic representations with little change to the core algorithm.
- Weaknesses: BERTopic assumes each document contains a single topic, although soft-clustering probabilities partially approximate topic distributions.This proxy does not account for multiple topics during training.
- Weaknesses: BERTopic’s topic representations remain bag-of-words based, so related words may become redundant and fail to directly encode contextual relationships.The authors note that maximal marginal relevance could address this issue but did not explore it.
8 Conclusion
The paper presents BERTopic, which combines language-model document embeddings, clustering, and class-based TF-IDF topic representations. Experiments suggest coherent language patterns and competitive, stable performance across varied tasks.
- Conclusion: BERTopic extends cluster-based topic modeling with state-of-the-art language models and class-based TF-IDF topic representations.Separating clustering from topic representation adds flexibility and ease of use.
- Conclusion: BERTopic is evaluated through topic coherence studies, running-time analyses, and other experiments.The paper provides an in-depth analysis spanning classical coherence measures and computational efficiency.
- Conclusion: BERTopic learns coherent language patterns and demonstrates competitive, stable performance across a variety of tasks.