Source-linked AI summary
Integrating Document Clustering and Topic Modeling
Pengtao Xie, Eric P. Xing
TL;DR
Document clustering and topic modeling are closely related tasks, but separate treatment does not allow them to mutually promote one another. The paper proposes MGCTM, a unified generative model that jointly discovers latent groups and local and global topics using variational inference. Experiments on two datasets show simultaneous clustering and topic modeling, with improved clustering and more coherent, differentiated topics.
Problem
Separate clustering and topic modeling do not allow the tasks to mutually promote each other, despite their close relationship and complementary information.
Method
MGCTM jointly infers latent group membership and topic variables in a generative model combining cluster-specific local topics with global topics shared across groups.
Results
Experiments on two datasets show improved clustering through topic modeling and more coherent, group-specific and group-independent topics through clustering.
Takeaways & Limitations
Joint modeling supports simultaneous document clustering and multi-grain topic extraction within one coupled framework.
Takeaways & Limitations
The authors identify extending MGCTM to semi-supervised clustering with incomplete external knowledge about likely same-cluster document pairs as future work.
Abstract
from arXiv · showhide
Document clustering and topic modeling are two closely related tasks which can mutually benefit each other. Topic modeling can project documents into a topic space which facilitates effective document clustering. Cluster labels discovered by document clustering can be incorporated into topic models to extract local topics specific to each cluster and global topics shared by all clusters. In this paper, we propose a multi-grain clustering topic model (MGCTM) which integrates document clustering and topic modeling into a unified framework and jointly performs the two tasks to achieve the overall best performance. Our model tightly couples two components: a mixture component used for discovering latent groups in document collection and a topic model component used for mining multi-grain topics including local topics specific to each cluster and global topics shared across clusters.We employ variational inference to approximate the posterior of hidden variables and learn model parameters. Experiments on two datasets demonstrate the effectiveness of our model.
1 INTRODUCTION
Document clustering and topic modeling can mutually benefit one another, motivating MGCTM, a unified model that jointly discovers document groups and multi-grain topics. Variational inference and experiments on two datasets support the model’s simultaneous clustering and topic-extraction capability.
- Motivation: Topic modeling projects documents into a semantic topic space that can improve clustering beyond raw term features.Topic models group synonymous words into topics and reduce noise in document similarity.
- Motivation: Document clustering enables local topics specific to each group and global topics shared across the collection.Local topics describe group-specific content, while global topics capture common semantics and can remove background words.
- Problem: Performing clustering and topic modeling separately fails to let the two tasks mutually promote each other.Separate pipelines either cluster topic-space representations or use fixed cluster labels for topic modeling.
- Approach: MGCTM jointly infers cluster membership, document-topic distributions, and topics in a generative model combining local and global topics.Each document belongs to one latent group and mixes topics specific to that group with topics shared across groups.
- Evaluation: The model uses variational inference for posterior inference and parameter learning, and experiments on two datasets evaluate simultaneous clustering and topic extraction.The experiments target local topics specific to clusters and global topics shared across clusters.
2 RELATED WORK
Related work covers conventional document clustering, topic-model-based clustering, probabilistic topic modeling, and models using category or fine-grained topic structures. These approaches motivate jointly modeling unknown clusters with local and shared topics.
- Document clustering: Document clustering organizes similar documents into groups and supports applications including organization, browsing, summarization, and classification.K-means and spectral clustering are widely used for document grouping.
- Document clustering: Matrix-factorization methods transform documents from term space into a lower-dimensional latent space to reduce similarity noise.The surveyed paradigms include LSI, NMF, and Concept Factorization.
- Document clustering: LDA and PLSA have been used for clustering by treating each topic as a cluster and assigning documents using their topic-proportion vectors.A document is assigned to the cluster corresponding to its largest topic proportion.
- Topic modeling: Topic models represent documents as mixtures of latent topics, with each topic modeled as a multinomial distribution over the vocabulary.Their purpose is to model texts and identify latent semantics in document collections.
- Joint and fine-grained models: Prior hierarchical and fine-grained models use category-specific, background, general, or ideology-specific topics, but some settings assume observed labels.The paper focuses on unsupervised clustering where cluster labels are unknown.
3 MULTI-GRAIN CLUSTERING TOPIC MODEL
MGCTM jointly performs document clustering and topic modeling by coupling a mixture component for latent groups with a topic model for local and global topics. It uses variational inference to estimate latent variables and learn parameters.
- Multi-grain topics: Local topics capture group-specific semantics, while global topics model universal semantics shared across all groups.The model gives each group its own local topic set and shares one global topic set across groups.
- Generative process: Each document is assigned to a group and represents words through either group-specific local topics or collection-wide global topics.A Bernoulli variable selects between local and global topic generation for each word.
- Model overview: MGCTM combines a mixture model for discovering document groups with a topic model for extracting group-specific and shared topics.Clustering estimates mixture-component variables, while topic modeling infers local and global topic variables.
- Joint learning: Joint inference optimizes latent variables from both components simultaneously, unlike sequential approaches that hold one component fixed.The paper argues that joint optimization can achieve at least the objective value obtained when either component is optimized with the other fixed.
- Model distinction: Compared with cluster-based topic models sharing one topic set, MGCTM allocates separate topic sets to groups.This design explicitly infers group-specific topics and avoids using one group's topics to generate documents from another group.
- Inference and learning: Variational inference approximates the intractable posterior using a parameterized distribution, with E-step updates followed by M-step lower-bound optimization.The variational parameters include multinomial, Beta, Dirichlet, and Bernoulli parameters; Dirichlet and Beta priors are optimized with Newton-Raphson.
4 EXPERIMENTS
Experiments on Reuters-21578 and 20-Newsgroups evaluate MGCTM for document clustering and multi-grain topic extraction. MGCTM generally outperforms or matches competing joint models while producing coherent global and cluster-specific local topics.
- Experimental Settings: MGCTM is evaluated for clustering and topic modeling on Reuters-21578 and 20-Newsgroups.Reuters-21578 uses 7,285 documents from 10 categories; 20-Newsgroups contains 18,370 documents from 20 groups.
- Experimental Settings: Clustering performance is assessed with accuracy (AC) and normalized mutual information (NMI), using the ground-truth category count as the input cluster number.The comparison includes K-means, normalized cut, factorization methods, topic-model-based methods, and MGCTM.
- Clustering Results: Topic-model-based clustering methods generally outperform K-means, normalized cut, and factorization-based methods on clustering metrics.The authors attribute this pattern to topic-model semantics providing more effective similarity information for forming coherent clusters.
- Clustering Results: Joint methods LDA+Naive, CTM, and MGCTM achieve much better results than LDA+Kmeans, which performs clustering and modeling separately.The result supports coupling clustering and topic modeling rather than treating them as two independent procedures.
- Clustering Results: MGCTM is generally better than or comparable with LDA+Naive and CTM because it models multiple local topics per cluster alongside cluster-independent global topics.LDA+Naive assigns one topic per cluster, while CTM does not distinguish cluster-specific from cluster-independent topics.
- Qualitative Evaluation: MGCTM global topics capture corpus-wide semantics, whereas local topics capture cluster-specific semantics such as computing, Middle East politics, space technology, and health.The qualitative evaluation uses 20-Newsgroups and reports global topics about news archives, time, and article writing.
5 CONCLUSIONS AND FUTURE WORK
The paper proposes MGCTM to jointly perform document clustering and topic modeling, showing that each task can improve the other. Future work will extend the model to semi-supervised clustering with incomplete external knowledge.
- MGCTM simultaneously performs document clustering and topic modeling in a unified framework.
- Experiments show that topic modeling improves clustering performance, while clustering produces more coherent and differentiated local and global topics.
- Future work will extend MGCTM to semi-supervised clustering settings.
- The extension must handle incomplete external knowledge indicating that some document pairs are likely to belong to the same cluster.