Source-linked AI summary
Top2Vec: Distributed Representations of Topics
Dimo Angelov
TL;DR
Topic modeling must organize large document collections despite traditional methods’ dependence on topic counts, stop-word handling, and bag-of-words representations. top2vec addresses these issues by jointly embedding documents and words to derive topic vectors. Its experiments show that top2vec finds topics more informative and representative of the corpus than LDA and PLSA.
Problem
Traditional topic models require topic-count assumptions, corpus- and language-specific stop-word filtering, and bag-of-words representations that ignore word semantics.
Method
top2vec jointly embeds document, word, and topic vectors so their distances represent semantic similarity.
Results
top2vec consistently finds topics that are more informative and representative of the training corpus than LDA and PLSA.
Takeaways & Limitations
top2vec automatically finds the number of topics and does not require stop-word lists, stemming, or lemmatization to learn informative topic words.
Takeaways & Limitations
The approach assumes a semantic space where document–word distance represents semantic association and similar documents are nearby.
Abstract
from arXiv · showhide
Topic modeling is used for discovering latent semantic structure, usually referred to as topics, in a large collection of documents. The most widely used methods are Latent Dirichlet Allocation and Probabilistic Latent Semantic Analysis. Despite their popularity they have several weaknesses. In order to achieve optimal results they often require the number of topics to be known, custom stop-word lists, stemming, and lemmatization. Additionally these methods rely on bag-of-words representation of documents which ignore the ordering and semantics of words. Distributed representations of documents and words have gained popularity due to their ability to capture semantics of words and documents. We present $\texttt{top2vec}$, which leverages joint document and word semantic embedding to find $\textit{topic vectors}$. This model does not require stop-word lists, stemming or lemmatization, and it automatically finds the number of topics. The resulting topic vectors are jointly embedded with the document and word vectors with distance between them representing semantic similarity. Our experiments demonstrate that $\texttt{top2vec}$ finds topics which are significantly more informative and representative of the corpus trained on than probabilistic generative models.
1 Introduction
The introduction frames topic modeling as a way to organize large text collections, then motivates top2vec as a distributed-representation alternative to traditional probabilistic models. It presents topics as continuous semantic structures and reports that top2vec produces more informative and representative topics than LDA and PLSA.
- Topic modeling discovers latent semantic structure in document collections and supports summarization, search, and grouping similar documents.
- Topics are treated as continuous combinations of weighted words, with each document having its own position in that topic continuum.
- Traditional Topic Modeling Methods: LDA and PLSA require the number of topics to be known or estimated, which is difficult for large or unfamiliar datasets.
- Traditional Topic Modeling Methods: Traditional models often require corpus- and language-specific stop-word filtering, while bag-of-words representations ignore word semantics.
- Distributed Representations of Topics: top2vec uses distributed representations to produce topic vectors without requiring stop-word lists, stemming, lemmatization, or prior knowledge of topic count.
- Distributed Representations of Topics: top2vec consistently finds topics that are more informative and representative of the training corpus than LDA and PLSA.
2.1 Create Semantic Embedding
This section constructs a semantic space by jointly embedding documents and words so that distances encode semantic association. It uses word2vec- and doc2vec-style prediction objectives to organize related words and documents near one another, then applies UMAP to preserve structure during dimension reduction.
- Create Semantic Embedding: A semantic embedding places similar documents close together, separates dissimilar documents, and places words near the documents they best describe.
- Create Semantic Embedding: DBOW uses document vectors to predict words in document context windows, enabling simultaneous learning of jointly embedded document and word vectors.
- Create Semantic Embedding: The skip-gram objective updates word and context vectors so semantically similar words become close while dissimilar words are separated.
- Create Semantic Embedding: In DBOW doc2vec, words from each document predict that document’s vector, bringing document vectors close to words occurring in them and away from unrelated words.
- Create Semantic Embedding: The learned word2vec and doc2vec space is treated as a continuous representation of topics, where points can be semantically described by nearby word vectors.
2.2 Find Number of Topics
top2vec identifies prominent topics by finding dense areas among document vectors in a semantic space. It reduces dimensionality with UMAP before using HDBSCAN to detect clusters and noise.
- Dense document areas in semantic space are interpreted as groups of highly similar documents sharing an underlying topic.
- The number of dense document areas is treated as the number of prominent topics found in the corpus.
- UMAP reduces high-dimensional document vectors so dense clusters can be found more efficiently and accurately.UMAP is chosen because it preserves local and global structure and scales to very large datasets.
- HDBSCAN clusters the UMAP-reduced document vectors while assigning noise labels to documents outside dense or prominent-topic regions.It is designed to handle variable-density clusters and noise.
- A minimum HDBSCAN cluster size of 15 gave the best results in the authors’ experiments.Larger values had a higher chance of merging unrelated document clusters.
2.3 Calculate Topic Vectors
top2vec calculates each topic vector from a dense document cluster, then represents the topic through nearby word vectors in the shared semantic space. Common words are generally not selected as topic words, removing the need for stop-word removal.
- Centroids, geometric means, and confidence-weighted alternatives produced very similar topic vectors in the authors’ experiments.
- HDBSCAN noise documents are excluded from the centroid used to calculate a topic vector.
- Each dense document cluster produces a topic vector by calculating the centroid of its document vectors.The centroid is the arithmetic mean of the vectors in the cluster.
- Topic words are the word vectors nearest to the topic vector because proximity represents semantic similarity.These words summarize the common topic shared by documents in the dense area.
- Common words are rarely nearest to topic vectors, so top2vec does not require stop-word removal.
2.4 Topic Size and Hierarchical Topic Reduction
top2vec measures topic size by assigning each document to its nearest topic vector. Its continuous topic representation also supports hierarchical reduction by repeatedly merging smaller topics with their most semantically similar neighbors.
- Each document belongs to the topic whose vector is most semantically similar to its document vector.
- Topic size is measured by the number of documents assigned to that topic.
- Hierarchical reduction can produce any number of topics below the initial count by iteratively merging the smallest topic into its nearest topic.
- Each merge uses a topic-size-weighted arithmetic mean of the two topic vectors, followed by recalculation of topic sizes.
- The reduction procedure biases the result toward larger topics, which are treated as more representative of the corpus.
3 Results
The evaluation measures topic usefulness by mutual information between documents and topic words, adapting the calculation to top2vec’s continuous topic space. Across 20 Newsgroups and Yahoo Answers, top2vec’s topic words were more informative and its assignments more representative than LDA and PLSA.
- Topic information gain: Mutual information measures the information gained about documents when they are described by their topic words.For top2vec, documents are grouped by their nearest topic vector before evaluating the words nearest to that vector.
- Topic information gain: A greater information-gain value indicates that topic words are more informative of their corresponding documents.Common words receive lower values because any specific document is unlikely given a very common word.
- Comparative results: Top2vec consistently provided more information than LDA and PLSA across topic sizes and up to the top 1000 topic words on 20 Newsgroups and Yahoo Answers.The comparison used the same documents and vocabulary for all models, including evaluations where stop-words were filtered from LDA and PLSA.
- Comparative results: For most topic sizes, top2vec’s top 20 words conveyed as much information as LDA and PLSA’s top 100 words on 20 Newsgroups.This comparison summarizes the information-gain curves while preserving the same evaluation corpus and vocabulary.
- Corpus representation: LDA’s high-information topics for encrypted or source-code messages represented only 23 of 18,831 messages, highlighting a coverage advantage for top2vec’s automatic topic discovery.Top2vec found a topic for that small message group before hierarchical reduction merged it because of its small size.
- Corpus representation: Top2vec’s 20-topic assignments nearly matched the 20 Newsgroups labels, while its top three words were very informative of each group’s topic.The corresponding Yahoo Answers visualization likewise showed near-matching assignments to its 10 topic labels and informative top-three words.
4 Discussion
Top2Vec represents topics in a continuous semantic space and compares words, documents, and topics by distance. The authors report that it produces more informative and representative topics than LDA and PLSA while automatically finding the topic count and avoiding stop-word lists.
- Model and representations: Top2Vec jointly embeds documents and words, then finds topic vectors in dense regions of the semantic space.The semantic space supports continuous topic representations and topic-size estimation.
- Evaluation: Topic information gain evaluates both the informativeness of topic words and the assignment of topics to documents.The measure uses mutual information to quantify information gained about documents from their topic words.
- Evaluation: Top2Vec consistently finds topics more informative and representative of the corpus than LDA and PLSA across topic sizes and numbers of top topic words.The comparison covers models trained on the 20 News Groups and Yahoo Answers datasets.
- Advantages over traditional models: Unlike LDA and PLSA, Top2Vec automatically finds the number of topics and does not require stop-word lists.Its distributed word representations also address limitations of bag-of-words representations that ignore word semantics.
- Advantages over traditional models: Top2Vec identifies informative topic words through document-word embedding rather than reconstructing document word distributions with accurate proportions.Words near a topic vector are tied to the documents from which the topic vector was calculated.
- Implementation: The Top2Vec code is available as an open-source project.The paper provides an implementation for the described model.