Source-linked AI summary
Topic Modeling over Short Texts by Incorporating Word Embeddings
Jipeng Qiang, Ping Chen, Tong Wang, Xindong Wu
TL;DR
Short texts provide limited word co-occurrence, making topic discovery difficult for established models. ETM combines embedding-guided pseudo-text aggregation with Markov Random Field topic regularization; experiments report more coherent topics and better clustering distributions than baselines on two datasets.
Problem
Short texts provide little word co-occurrence information, and existing topic models such as PLSA and LDA do not model them very well.
Source: §1 · p. 1Method
ETM aggregates short texts into pseudo-texts using word-embedding semantics, then applies a Markov Random Field regularizer that encourages similar words to share a topic.
Source: §1 · p. 2Results
On Tweet 2011 and Google News, ETM is reported to produce more prominent, coherent topics and significantly better topic distributions for clustering than baselines.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that ETM is effective qualitatively and quantitatively against state-of-the-art methods on two real-world short-text datasets.
Source: §5 · p. 9The model description specifies a corpus with K topics and L pseudo-texts over V vocabulary words, with α and β as hyperparameters.
Source: §3.3.1 · p. 4
Abstract
from arXiv · showhide
Inferring topics from the overwhelming amount of short texts becomes a critical but challenging task for many content analysis tasks, such as content charactering, user interest profiling, and emerging topic detecting. Existing methods such as probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA) cannot solve this prob- lem very well since only very limited word co-occurrence information is available in short texts. This paper studies how to incorporate the external word correlation knowledge into short texts to improve the coherence of topic modeling. Based on recent results in word embeddings that learn se- mantically representations for words from a large corpus, we introduce a novel method, Embedding-based Topic Model (ETM), to learn latent topics from short texts. ETM not only solves the problem of very limited word co-occurrence information by aggregating short texts into long pseudo- texts, but also utilizes a Markov Random Field regularized model that gives correlated words a better chance to be put into the same topic. The experiments on real-world datasets validate the effectiveness of our model comparing with the state-of-the-art models.
1. INTRODUCTION
Short texts provide little word co-occurrence information, making established topic models and existing short-text strategies difficult to apply. ETM uses embedding-based semantic relationships to aggregate short texts and regularize topic assignments, and is reported to improve topic coherence and clustering distributions.
1. INTRODUCTION
- Short texts offer limited word co-occurrence, less discriminative word frequencies, and restricted context for resolving ambiguous words, challenging topic discovery.LDA does not work very well on short texts.Source: §1 · p. 1
- Existing strategies either assume each short text has one topic or aggregate texts using heuristic ties, but these approaches leave co-occurrence or context limitations unresolved.Texts without shared words can be difficult to aggregate, even when they are semantically related.Source: §1 · p. 1
- ETM uses word embeddings, whose vector distances are treated as semantically meaningful, to incorporate external word-correlation knowledge into short-text topic modeling.The passage illustrates semantic vector relationships with the King–man+woman and Queen example.Source: §1 · p. 2
- ETM builds word embeddings, aggregates short texts into pseudo-texts using semantic knowledge, then applies a Markov Random Field regularizer to encourage similar words to share topics.The regularization is soft and topic-dependent; the paper says this helps identify ambiguous-word senses and improve coherence.Source: §1 · p. 2
- On Tweet 2011 and Google News, ETM is reported to find more prominent, coherent topics and significantly better topic distributions for clustering than baselines.The supplied results passage gives no numerical comparisons.Source: §1 · p. 2
2. RELATED WORK
Prior topic-modeling work spans long-text models and short-text approaches that enrich or aggregate sparse texts, or assume one topic per text. The related work motivates ETM by noting that these short-text strategies do not resolve limited word co-occurrence or generalize to texts lacking context ties.
2.1 Long Text Topic Modeling
- Long-text topic models commonly represent documents as mixtures of topics, while knowledge-based variants encode word relations but are described as performing poorly on short texts.The cited examples include PLSA, LDA, must-link/cannot-link constraints, and Markov Random Field regularization.Source: §2.1 · p. 2
2.2 Short Text Topic Modeling
- Some short-text approaches enrich texts with related external data, but they depend on that data being closely related and may require expensive or unavailable collection.Source: §2.2 · p. 2
- Other approaches aggregate texts using user, hashtag, or clustering ties, or assume one topic per text; these strategies do not resolve sparse co-occurrence or readily fit texts lacking context information.The paper proposes incorporating word-correlation knowledge from embeddings as motivation for its approach.Source: §2.2 · pp. 2–3
3. ALGORITHM
ETM builds word embeddings, groups short texts into longer pseudo-texts, and infers topics with an MRF-regularized model. Its word-distance-based grouping and topic-assignment dependencies incorporate semantic relationships into the modeling pipeline.
3. ALGORITHM
- ETM combines word embeddings, WMD-based short-text clustering, and MRF-regularized topic inference to model short texts as long pseudo-texts.The method has three stages: build embeddings, aggregate short texts, then infer topics from the pseudo-texts.Source: §3 · p. 3
3.1 Word Embeddings
- Word embeddings provide vector representations that can capture semantic relationships, including analogies expressed through vector operations.The paper describes Word2Vec’s shallow neural network and Glob2Vec’s combination of global co-occurrence counts with local context windows.Source: §3.1 · p. 3
3.2 Short Text Clustering
- WMD measures text distance by the minimum cost of transporting word weights between texts, using embedding-based word distances.The paper relaxes one flow constraint to reduce the stated time complexity from O(m^3logm) to O(mlogm), then uses K-means to form pseudo-texts.Source: §3.2 · pp. 3–4
3.3 Topic Inference
- ETM infers topics from the long pseudo-texts using an MRF-regularized model and collapsed Gibbs sampling.The model extends LDA by connecting topic assignments for words whose embedding-based distances fall below a threshold.Source: §3.3 · p. 4, §3.3.1 · pp. 4–5
3.3.1 Model Descritpion
- The MRF’s binary potential favors matching topic assignments for connected, semantically similar words, while λ controls its tradeoff with the unary potential.When λ=0, the MRF model reduces to LDA; otherwise, an assignment depends on both the pseudo-text’s topic distribution and similar words’ assignments.Source: §3.3.1 · p. 4
3.3.2 Parameter Estimation
- Collapsed Gibbs sampling estimates topic assignments by integrating out φ and θ; assignment counts then yield topic-word and pseudo-text topic distributions.The method also describes deriving topic assignments for each short text from the assignments in its long pseudo-text.Source: §3.3.1 · p. 5
4. EXPERIMENTS
Across Tweet2011 and Google News, ETM produced more coherent topics and better clustering results than the baseline models. The experiments evaluate topic quality and clustering on both short-text datasets.
4.1 Datasets Description and Setup
- The evaluation used Tweet2011 tweets and Google News article titles, with Glob2Vec embeddings selected for ETM and MRF-LDA.The Google News snapshot contained 6,974 headlines across 134 categories.Source: §4.1 · p. 6
4.2.1 Qualitative Evaluation
- In qualitative evaluations, ETM learned more coherent topics than all baseline models on Tweet2011 and Google News; it also had fewer noisy, meaningless words on Tweet2011.The Google News examples cover four events on April 27, 2015: Nepal earthquake, Iran nuclear, Indonesia Bali, and Yemen airstrikes. On Tweet2011, ETM also recognized topic words with only a few occurrences.Source: §4.2.1 · pp. 6, 8
4.2.2 Quantitative Evaluation
- ETM significantly outperformed the baselines in coherence on both datasets; Tweet2011 average CM was 72.5%.Tables 3 and 4 report coherence measures for Tweet2011 and GoogleNews, respectively.Source: §4.2.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
4.2.3 Short Text Clustering
- ETM significantly outperformed long- and short-text topic models in clustering on both datasets, with lower NMI standard deviations than all other methods.The comparison used mean and standard deviation over 20 runs per model and dataset.Source: §4.2.3 · p. 8, Table 5 · p. 8
5. CONCLUSION
ETM discovers latent topics from short texts by aggregating them into pseudo-texts using semantic knowledge from word embeddings, then applying an MRF-regularized model. Experiments on two real-world short-text datasets corroborate its effectiveness qualitatively and quantitatively over state-of-the-art methods.
5. CONCLUSION
- ETM aggregates short texts into long pseudo-texts by incorporating semantic knowledge from word embeddings.Source: §5 · pp. 8–9
- Experiments on two real-world short-text datasets corroborate ETM’s effectiveness qualitatively and quantitatively over state-of-the-art methods.The conclusion reports both forms of evidence without specifying individual metrics.Source: §5 · p. 9
- ETM infers topics from the pseudo-texts using a Markov Random Field regularized model.Source: §5 · p. 9