PaperlayerView exact revision on arXiv

Source-linked AI summary

Word Network Topic Model: A Simple but General Solution for Short and Imbalanced Texts

Yuan Zuo, Jichang Zhao, Ke Xu

arXiv:1412.5404v1cs.CLcs.IR

TL;DR

Short texts are difficult for conventional topic models because sparse context and imbalanced topic distributions hinder reliable topic discovery. WNTM models word co-occurrence networks and learns topic distributions for words, improving semantic density without excessive complexity. Experiments report stronger performance than baselines and potential for early rare-topic and emerging-event detection, with complexity becoming unacceptable for normal texts under some settings.

  • Problem

    Sparse context and imbalanced topic distributions make conventional LDA-like topic modeling difficult for short texts.

  • Method

    WNTM learns topic distributions for words from word co-occurrence networks rather than topic distributions for documents.

  • Results

    WNTM outperforms baseline methods on topic coherence, word similarity, and document classification across short and normal texts, with stronger rare-topic detection in extremely imbalanced texts.

  • Takeaways & Limitations

    WNTM may support early detection of rare, newly emerging topics or unexpected social-media events.

  • Takeaways & Limitations

    For normal texts, WNTM’s time complexity becomes unacceptable when the average document length parameter can be large.

Abstract

from arXiv · show

The short text has been the prevalent format for information of Internet in recent decades, especially with the development of online social media, whose millions of users generate a vast number of short messages everyday. Although sophisticated signals delivered by the short text make it a promising source for topic modeling, its extreme sparsity and imbalance brings unprecedented challenges to conventional topic models like LDA and its variants. Aiming at presenting a simple but general solution for topic modeling in short texts, we present a word co-occurrence network based model named WNTM to tackle the sparsity and imbalance simultaneously. Different from previous approaches, WNTM models the distribution over topics for each word instead of learning topics for each document, which successfully enhance the semantic density of data space without importing too much time or space complexity. Meanwhile, the rich contextual information preserved in the word-word space also guarantees its sensitivity in identifying rare topics with convincing quality. Furthermore, employing the same Gibbs sampling with LDA makes WNTM easily to be extended to various application scenarios. Extensive validations on both short and normal texts testify the outperformance of WNTM as compared to baseline methods. And finally we also demonstrate its potential in precisely discovering newly emerging topics or unexpected events in Weibo at pretty early stages.

1 Introduction

Short texts provide valuable signals but challenge conventional topic models through sparse context and imbalanced topic distributions. WNTM addresses both issues by learning topics in word co-occurrence space and shows strong performance across evaluated tasks.

  • Motivation: Short texts are increasingly prevalent and support tasks including content analysis, query suggestion, document classification, and text clustering.Twitter alone is described as generating almost 500 million tweets daily.
  • Challenges: LDA-like models rely on document-level word co-occurrence, making them sensitive to short document length and skewed topic distributions.Sparse word counts hinder accurate word relationships, while majority topics can overshadow rare topics.
  • Related approaches: Existing solutions aggregate short texts, use external data, or modify LDA for specific scenarios, limiting generality or flexibility.The biterm topic model works on short texts but is not LDA-based and remains constrained by its specialized formulation.
  • Proposed approach: WNTM learns topic components from word co-occurrence networks and topic distributions for words rather than documents.Its design targets sparse word-by-document space and less-skewed word-topic distributions while retaining a general framework.
  • Results: WNTM outperforms baseline methods on topic quality, word similarity, and document categorization across short and normal texts.It is particularly stronger than LDA for rare-topic detection in extremely imbalanced texts and uses standard Gibbs sampling.

2 Related works

Prior work addresses sparse or imbalanced texts using aggregation, auxiliary information, specialized models, or informative priors. The paper positions WNTM as a simple, general solution that handles sparsity and imbalance simultaneously without external knowledge.

  • General topic models: Probabilistic topic models such as PLSA and LDA have been widely applied, while many variants add properties such as time, social relationships, or authorship.Most such extensions target normal texts with additional structure.
  • Sparse texts: Short-text methods increase data density through tweet aggregation, similarity measures, or transfer learning from auxiliary long-text data.These approaches rely on additional information or transformations of the original short texts.
  • Imbalanced texts: Imbalanced-text methods use prior knowledge, lexical constraints, labels, or supervised topic-model formulations to address skewed document-topic distributions.Examples include Dirichlet forest priors, lexical knowledge, labels, and supervised topic models.
  • Positioning: WNTM targets sparsity and imbalance simultaneously through a simple, general framework without exploiting external knowledge.The paper describes this as a comparatively underexplored problem.

3 Word network topic model

WNTM replaces sparse document-level topic modeling with topic learning in a word co-occurrence network, representing each word’s contextual neighborhood as a pseudo-document. It applies Gibbs sampling to latent word groups and derives document topics from word-level topic proportions.

  • Word network motivation: WNTM addresses short-text sparsity and topic imbalance by learning topics in word-word space rather than word-by-document space.The model uses dense co-occurrence structure while preserving semantically compact groups associated with rare topics.
  • Word co-occurrence network: A word co-occurrence network represents words as nodes and connects words that co-occur within a document or fixed-size sliding window.WNTM uses a sliding window of size 10 and weights edges by repeated co-occurrence counts.
  • Word network topic model: WNTM treats latent word groups in the network as topics because frequently co-occurring words form closely connected semantic groups.The model converts each word’s adjacent-word list into a pseudo-document, with repeated adjacent words reflecting edge weights.
  • Word network topic model: WNTM uses LDA-style Gibbs sampling to generate each word’s adjacent-word list from latent word groups and their word distributions.Each word has a latent group distribution over its adjacent-word list, while each group has a multinomial distribution over vocabulary words.
  • Inferring document topics: Document topic proportions are inferred by combining each word’s learned topic proportions with the document’s empirical word distribution.The empirical word distribution uses word frequency divided by document length, making the procedure straightforward to implement.

4 Complexity analysis and word network re-weighting

WNTM’s network representation increases computational cost relative to LDA because pseudo-document lengths depend on network degree. The paper therefore proposes re-weighting edges to reduce weighted degrees and lower time and space consumption.

  • Complexity analyses: WNTM has time complexity O(NpKgLp), whereas LDA has time complexity O(NdKzLd).Here Np is vocabulary size, Kg is the number of latent word groups, and Lp is average pseudo-document length.
  • Complexity analyses: When Kz equals Kg, WNTM’s time and memory costs are o(c2) times LDA’s under the stated network assumptions.The paper considers this acceptable for short texts but unacceptable for normal texts when the window size c is large.
  • Word network re-weighting: The dominant tunable factor is average pseudo-document length because vocabulary size fixes the number of pseudo-documents.Reducing weighted node degrees therefore reduces NpLp, the quantity governing both time and space consumption.
  • Word network re-weighting: The re-weighting algorithm computes each node’s degree and activity, then divides each edge weight by the activity of its lower-degree endpoint.The resulting network has reduced weighted degree and shorter average pseudo-documents.

5 Experiments

The experiments evaluate WNTM against baseline topic models on short and normal texts using topic quality, word similarity, and document classification. The study uses different baselines and computing environments for the data scales considered.

  • Evaluation design: The evaluation measures topic quality, word similarity, and document classification on both short and normal real-world texts.Short-text experiments compare WNTM with LDA and BTM, while normal-text experiments omit BTM because of its high time complexity.
  • Experimental setup: Most experiments run on a Windows Server with an Intel Xeon 2.40GHz CPU and 12G memory, while Wikipedia experiments use a 13-node Linux cluster.Each cluster node contains 2 Intel Xeon 2.27Hz CPUs and 12 GB memory.
  • Experimental setup: All experiments use 100 topics, a WNTM sliding-window length of 10, and 2,000 Gibbs-sampling iterations.The implementations and Dirichlet parameters differ across LDA, WNTM, BTM, and PLDA as specified for each setting.

5.1 Evaluation of the topic quality

The evaluation uses topic coherence rather than perplexity to assess interpretability, comparing WNTM with LDA and BTM on short micro-blogs and with LDA on normal Wikipedia texts. WNTM achieves higher coherence on short texts, while its advantage on normal texts depends on the number of top words evaluated.

  • 5.1.1 Topic coherence: Topic coherence measures semantic similarity among a topic’s high-probability words, with higher scores indicating more interpretable topics.The study uses average coherence across topics rather than perplexity because WNTM does not model document generation and perplexity may not track interpretability.
  • 5.1.2 Topic coherence on short texts: On 189,223 Weibo micro-blogs, WNTM’s average topic coherence is significantly higher than LDA and BTM across top-word settings T = 5–20.The improvement over both baselines is statistically significant with p-value < 0.001 by t-test.
  • 5.1.3 Topic coherence on normal texts: On Wikipedia texts, WNTM is slightly more coherent than LDA for T = 5 and 10 but slightly less coherent for T = 20.The comparison uses 71,986 documents with an average document length of 423.5 tokens.
  • 5.1.3 Topic coherence on normal texts: The short-text advantage is attributed to learning topics in dense word-word space, whereas normal documents already provide rich document-level context for LDA.For normal texts, the two representational spaces therefore make little difference in topic quality.

5.2 Word similarity tasks

The word similarity experiments evaluate semantic representations using ranked correlation between model-derived and human word-similarity judgments. WNTM outperforms LDA on both short Weibo texts and normal Wikipedia texts, while also surpassing BTM on Weibo under cosine similarity.

  • Semantic representation: WNTM represents each word with its learned topic-distribution row vector, whereas LDA and BTM derive word representations from conditional topic distributions.For WNTM, the semantic representation is directly available from matrix Θ after learning topic distributions over words.
  • Evaluation setup: Word similarity is evaluated by comparing model ratings with human ratings using ranked correlation, where higher correlation indicates better word semantic modeling.Jensen-Shannon divergence and cosine similarity are used to compare word representations.
  • Short-text results: On Weibo short texts, WNTM and BTM significantly outperform LDA with both Jensen-Shannon and cosine-based similarity measurements.WNTM performs similarly to BTM on Jensen-Shannon similarity but significantly outperforms BTM on cosine similarity.
  • Overall finding: The results indicate that WNTM produces more effective word semantic representations than LDA across short and normal texts.The short-text comparison additionally reports WNTM as more reliable than BTM.
  • Normal-text results: WNTM remains superior to LDA on Wikipedia normal texts for both similarity measurements, although the performance gap shrinks as document-level sparsity disappears.WNTM also has much lower deviations, indicating more stable results, despite similar topic coherence to LDA.

5.3 Document classification

The section evaluates WNTM and baseline models on news-title and news-content classification, including highly imbalanced settings. WNTM outperforms LDA across short and normal texts and is especially effective for rare-topic identification and early emerging-event detection.

  • News corpus evaluation: The experiments classify news titles and contents using topic representations, with titles averaging 5.5 tokens and reports averaging 175.9 tokens.The evaluation uses 10-fold cross-validation and LIBLINEAR classification on 100-topic models.
  • News corpus evaluation: WNTM and BTM outperform LDA on classification of both short and normal texts.The authors attribute the larger short-text divergence to data sparsity affecting LDA more strongly, while WNTM remains less variable.
  • News corpus evaluation: WNTM continues to outperform LDA on normal texts, where LDA’s classification becomes acceptable because the data are no longer sparse.This result indicates stronger document-resemblance recognition for WNTM in both sparse and less-sparse settings.
  • Imbalanced document classification: WNTM improves identification of rare labels in imbalanced news contents, whereas LDA nearly misclassifies the rare “House” label and also confuses “Culture” and “Health”.The confusion patterns are associated with classes having relatively few documents.
  • Imbalanced document classification: WNTM identifies the emerging “MH370” topic when at least 30 related micro-blogs are injected, while LDA requires at least 50.The experiment uses 10,000 unrelated pre-event Weibo posts and counts detection across repeated Gibbs-sampling runs.

6 Conclusions

The conclusion presents WNTM as a simple, general approach for topic modeling in short and imbalanced texts. Experiments across short and normal texts show advantages over baseline methods, especially for rare topics and early social-media event detection, while identifying network construction as future work.

  • Conclusion: WNTM explores topics from word co-occurrence networks to alleviate sparsity and topic-document heterogeneity at acceptable cost.The approach is intended as a general framework for short and imbalanced texts.
  • Conclusion: Experiments on short and normal texts show that WNTM outperforms baseline methods in topic coherence, word similarity, and document classification.The conclusion summarizes results across all three evaluation tasks.
  • Conclusion: WNTM’s ability to capture rare topics supports its potential for detecting newly emerging topics or unexpected social-media events at early stages.The conclusion frames this as a potential application rather than an unrestricted guarantee.
  • Conclusion: Future work includes studying how alternative word co-occurrence network construction methods affect topics and using neighboring words within semantic distance.These directions are presented as open research questions.
Loading 1412.5404v1…