PaperlayerView exact revision on arXiv

Source-linked AI summary

Short Text Topic Modeling Techniques, Applications, and Performance: A Survey

Qiang Jipeng, Qian Zhenyu, Li Yun, Yuan Yunhao, Wu Xindong

arXiv:1904.07695v1cs.IRcs.CL

TL;DR

Short texts provide limited word co-occurrence information, challenging traditional topic models despite the need for semantic understanding. The survey reviews and categorizes short-text methods, develops STTM, and evaluates surveyed approaches across real datasets and tasks. It reports dataset-dependent performance patterns, with some DMM-based embedding methods outperforming others on particular datasets, while evaluation remains an open problem.

  • Problem

    Short texts offer limited word co-occurrence information, making traditional PLSA and LDA less effective for inferring latent topics.

  • Method

    The survey presents a three-category taxonomy, develops the STTM open-source Java library, and evaluates surveyed methods across multiple tasks and datasets.

  • Results

    Performance is dataset dependent; DMM-based methods using word embeddings outperform others especially on Tweet and GoogleNews, while GPU-PDMM performs best except on SearchSnippets.

  • Takeaways & Limitations

    STTM consolidates surveyed short-text topic modeling methods and supports analysis, inference, evaluation, classification, and clustering.

  • Takeaways & Limitations

    Evaluating short-text topic models remains an open problem because conventional metrics can be problematic for short texts and likelihood may not indicate topic quality.

Abstract

from arXiv · show

Analyzing short texts infers discriminative and coherent latent topics that is a critical and fundamental task since many real-world applications require semantic understanding of short texts. Traditional long text topic modeling algorithms (e.g., PLSA and LDA) based on word co-occurrences cannot solve this problem very well since only very limited word co-occurrence information is available in short texts. Therefore, short text topic modeling has already attracted much attention from the machine learning research community in recent years, which aims at overcoming the problem of sparseness in short texts. In this survey, we conduct a comprehensive review of various short text topic modeling techniques proposed in the literature. We present three categories of methods based on Dirichlet multinomial mixture, global word co-occurrences, and self-aggregation, with example of representative approaches in each category and analysis of their performance on various tasks. We develop the first comprehensive open-source library, called STTM, for use in Java that integrates all surveyed algorithms within a unified interface, benchmark datasets, to facilitate the expansion of new methods in this research field. Finally, we evaluate these state-of-the-art methods on many real-world datasets and compare their performance against one another and versus long text topic modeling algorithm.

1 INTRODUCTION

Short-text topic modeling addresses sparse word co-occurrences that limit traditional models, supporting semantic analysis across applications. The survey organizes methods, provides STTM, and compares techniques across applications.

  • Traditional PLSA and LDA discover latent semantic structure from document-level word co-occurrences, but short texts provide limited such information.
  • External knowledge, metadata, and aggregation strategies add word co-occurrences, but auxiliary information may be unavailable or costly.
  • Short texts lack enough word co-occurrence information, making latent topic inference difficult.
  • The survey categorizes methods into Dirichlet multinomial mixture, global word co-occurrences, and self-aggregation approaches.
  • The authors develop STTM, an open-source Java library integrating surveyed algorithms through a uniform interface with evaluation and application modules.
  • The survey analyzes performance across applications and conducts comprehensive comparative evaluations on common datasets.

2 DEFINITIONS

Short-text topic modeling seeks topic-word and document-topic representations for a corpus with predefined topics. The paper defines these representations probabilistically and identifies the main modeling tasks.

  • A short-text corpus contains N documents, a vocabulary W of size V, and K predefined latent topics.
  • A topic φ is a multinomial distribution over the vocabulary W.
  • A document topic representation θ_d is a multinomial distribution over K topics.
  • Topic modeling aims to learn K salient topics and the topic representation of every document.
  • Dirichlet priors smooth topic mixtures and topic-word distributions, alleviating PLSA overfitting as topic and vocabulary sizes increase.
  • Short-text topic modeling primarily learns topic word representations φ and sparse document topic representations θ.

3 ALGORITHMIC APPROACHES: A TAXONOMY

Short-text topic modeling evolved from adapting traditional models with external information toward methods designed specifically for sparse short texts. The survey presents this progression through aggregation, metadata, and new model families.

  • The survey subsequently categorizes newer short-text topic modeling techniques into DMM-based, global-word-co-occurrence, and self-aggregation methods.
  • Traditional PLSA and LDA capture document-level word co-occurrences, whose scarcity causes performance degradation on short texts.
  • Early approaches incorporated external knowledge or metadata to enhance short-text topic inference.
  • Since 2010, researchers merged short texts into long pseudo-documents before applying traditional topic models.
  • Aggregation strategies used users, hashtags, timestamps, and named entities as bases for merging short texts.

3.2 A Taxonomy of Short Text Topic Modeling Methods

The survey proposes a taxonomy with three broad categories of short-text topic modeling methods and describes representative approaches within each category.

  • The taxonomy divides approaches into Dirichlet Multinomial Mixture based, Global Word co-occurrences based, and Self-aggregation based methods.

3.3 Dirichlet Multinomial Mixture based Methods

Dirichlet multinomial mixture methods model short texts with simplified topic-generation assumptions and extend them using latent features, semantic reinforcement, or multiple-topic generation. Representative methods include GSDMM, LF-DMM, GPU-DMM, and GPU-PDMM.

  • DMM-based methods: DMM assumes each short text is sampled from one latent topic, fitting short texts better than LDA’s multiple-topic assumption.This simplification addresses the limited word co-occurrence information available in short texts.
  • GSDMM: GSDMM applies collapsed Gibbs sampling under the single-topic assumption and represents each topic with a topic feature vector.The topic feature vector aggregates word and document counts for efficient updates.
  • LF-DMM: LF-DMM combines Dirichlet multinomial generation with a latent feature model selected independently for each word through a binary indicator.The latent feature model uses pre-trained word vectors and a softmax function, while topic vectors are optimized with L-BFGS.
  • GPU-DMM: GPU-DMM reinforces semantically similar words during sampling when a word has strong ties with the sampled topic.It constructs promotion relationships from cosine similarity between pre-trained word embeddings and updates topic features through addible and deletable properties.

3.4 Global Word Co-occurrences based Methods

Global word co-occurrence methods address short-text sparsity by learning topics from corpus-level word relationships rather than relying only on individual documents. Representative approaches include BTM, which models biterms, and WNTM, which models a word co-occurrence network.

  • Sliding windows extract global word co-occurrences, with window size typically set to 10 when average document length exceeds 10.Shorter documents can be treated directly as sliding windows.
  • BTM: BTM treats any two words in a document as a biterm and infers topics over the resulting biterm collection.Each biterm’s two words share a topic-specific multinomial distribution.
  • BTM: BTM uses collapsed Gibbs sampling to reassign each biterm’s topic and update topic feature vectors through deletion and addition operations.The procedure estimates topic-word and topic-proportion distributions after iterative sampling.
  • WNTM: WNTM constructs an undirected word co-occurrence network whose nodes are words and whose edge weights count their co-occurrences.The network is built from words appearing together within sliding windows.
  • WNTM: WNTM converts each vertex’s adjacent-word list into a weighted pseudo-document and applies LDA to learn topic distributions for words.A pseudo-document’s length equals the source vertex’s degree, and the number of pseudo-documents equals the vocabulary size.
  • WNTM: In WNTM, document-word distributions are calculated from the learned topic-word distributions.

3.5 Self-aggregation based Methods

Self-aggregation methods reduce short-text sparsity by assigning or merging texts into longer pseudo-documents before or during topic inference. SATM and PTM integrate pseudo-document structure with topic modeling using sampling-based procedures.

  • Self-aggregation methods merge short texts into long pseudo-documents, usually producing far fewer pseudo-documents than original texts.The survey states |P|≪N for pseudo-documents P and short texts N.
  • SATM: SATM assumes each short text is sampled from an unobserved long pseudo-document and infers topics from those pseudo-documents.
  • SATM: SATM first estimates each pseudo-document’s probability given a short document using a mixture-of-unigrams model.
  • SATM: SATM then jointly samples pseudo-document and topic labels for each word, using pseudo-document and topic feature updates during Gibbs sampling.
  • PTM: PTM assumes each short text is sampled from one long pseudo-document and infers latent topics from the resulting pseudo-documents.A multinomial distribution models short texts over pseudo-documents.
  • PTM: PTM samples a pseudo-document, then samples each word’s topic from that pseudo-document before generating the word from the topic distribution.Collapsed Gibbs sampling estimates pseudo-document assignments for short texts.

4 APPLICATIONS

Short-text topic models support social-media content analysis across recommendation, classification, and event tracking. Applications use topics to represent content, identify interests, construct features, and follow evolving events.

  • Social-media topic modeling is used for content characterization, recommendation, text classification, and event tracking.
  • Content analysis: Twitter topic models discover substance and social topics, among other categories, from user-generated tweets.
  • Content recommendation: Recommendation systems match emerging Twitter topics with recent news or model content sources and user topic interests.
  • Text classification: Classification approaches either add topics learned from external corpora as features or use topic models to create discriminative features.
  • Text classification: Topic-model representations provide compact, dense, lower-dimensional vectors whose dimensions usually correspond to semantic topics.
  • Event tracking: Event-tracking models monitor popular events or topics as they evolve over time, including user-interest burstiness and time-slice updates.Some approaches use dynamic vocabularies or built-in update mechanisms.

5 A JAVA LIBRARY FOR SHORT TEXT TOPIC MODELING

STTM is an open-source Java library that unifies short-text topic-modeling algorithms with evaluation and application modules. Its architecture supports analysis, inference, evaluation, classification, and clustering workflows.

  • STTM integrates state-of-the-art short-text topic-modeling algorithms through a uniform, easy-to-use programming interface.
  • The library includes modules for evaluating and applying short-text topic-modeling algorithms.
  • STTM also integrates LDA and LF-LDA, enabling comparisons between long-text and short-text topic-modeling algorithms.
  • Its architecture covers the knowledge-discovery procedure from analysis and inference through evaluation, classification, and clustering.

6 EXPERIMENTAL SETUP

The experiments define datasets, model settings, and evaluation metrics for comparing nine short-text topic models. Evaluation covers classification, clustering, and topic coherence, while recognizing that short-text model evaluation remains unresolved.

  • Datasets: Experiments compare nine models on six datasets characterized by topic count, document count, document length, and vocabulary size.The datasets include SearchSnippets, StackOverflow, Biomedicine, Tweet, GoogleNews, and PascalFlickr.
  • Evaluation metrics: Short-text topic-model evaluation remains open because metrics suitable for long texts, including perplexity, may not reliably measure short-text quality.Held-out likelihood is not necessarily a good indicator of topic quality in short texts.
  • Evaluation metrics: Classification evaluates document-topic distributions with accuracy using a default linear-kernel LIBLINEAR SVM.Higher accuracy is treated as evidence that learned topics are more discriminative and representative.
  • Evaluation metrics: Clustering assigns each document its highest-probability topic and evaluates the resulting labels with Purity and NMI.The labels are compared against gold labels.
  • Evaluation metrics: Topic coherence scores the top topic words using PMI from a 10-word sliding window over more than one million English Wikipedia articles.This evaluation requires Wikipedia as an additional meta-document.

7 EXPERIMENTS AND ANALYSIS

Across classification, clustering, coherence, convergence, and efficiency experiments, performance varies by dataset and method family. DMM-based methods are generally strong, while self-aggregation methods often perform poorly and some embedding-based methods are computationally costly.

  • Classification Accuracy: DMM methods using word embeddings outperform other models in classification, especially on Tweet and GoogleNews.Retraining embeddings on domain-specific data may further improve performance on domain-specific datasets.
  • Classification Accuracy: BTM and GSDMM outperform LDA on every dataset, while GPU-PDMM performs best except on SearchSnippets.Other methods, including WNTM, remain highly dataset dependent.
  • Clustering: All models except SATM outperform LDA on clustering metrics, but performance remains highly dataset dependent.WNTM is strongest on several datasets yet performs poorly on PascalFlickr, while GPU-PDMM performs very well except on SearchSnippets.
  • Clustering: PTM outperforms SATM, WNTM outperforms BTM on Tweet and StackOverflow, and GSDMM leads other DMM methods on Biomedicine and SearchSnippets.These comparisons illustrate variation within each method family.
  • Topic Coherence: DMM methods achieve the best topic coherence on all datasets; LF-DMM leads four datasets, GPU-DMM leads StackOverflow, and GPU-PDMM leads PascalFlickr.The qualitative evaluation similarly finds LF-DMM produces coherent topics with fewer noisy words.
  • Convergence: DMM models converge within about 30 iterations, global-co-occurrence models within 60, and self-aggregation models are slowest with the lowest iterative performance.The comparison uses NMI while varying iterations from 2 to 2024.
  • Efficiency: LDA and DMM are most efficient, global-co-occurrence methods are slower than them but faster than most remaining models, and GPU-PDMM is slowest.Embedding-based models also have high initialization costs because of word-similarity computation.

8 CONCLUSION AND FUTURE WORK

The survey organizes short-text topic modeling into three method categories, packages the surveyed approaches and evaluation tasks in STTM, and compares them across six real datasets. It concludes that evaluation, visualization, and model selection remain open research directions.

  • Contributions: The survey covers DMM-based, global-word-co-occurrence, and self-aggregation methods.It characterizes the structures, properties, and challenges associated with these categories.
  • Contributions: STTM is an open-source Java library integrating surveyed methods with classification, clustering, and topic-coherence evaluation tasks.The library supports a unified research and evaluation workflow.
  • Contributions: The approaches are evaluated on six publicly available real datasets, with their strengths and weaknesses compared across tasks.The survey presents this comparison as part of its final evaluation.
  • Future work: Future work includes better topic and document visualization, evaluation metrics that match model use, and model checking for new corpora.The paper notes that existing metrics do not fully distinguish topics or evaluate every part of topic modeling.
Loading 1904.07695v1…