PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Topic Models with Latent Feature Word Representations

Dat Quoc Nguyen, Richard Billingsley, Lan Du, Mark Johnson

arXiv:1810.06306v1cs.CLcs.IRcs.LG

TL;DR

Small or short corpora provide limited evidence for learning reliable topic-to-word mappings. The paper integrates latent feature word representations trained on large external corpora into LDA and DMM, and reports improvements across topic coherence, document clustering, and document classification, especially for few or short documents.

  • Problem

    Small or short corpora can provide too little co-occurrence evidence for conventional topic models to estimate reliable document-to-topic and topic-to-word distributions.

  • Method

    The paper proposes LF-LDA and LF-DMM, mixing externally trained latent feature word representations with Dirichlet multinomial components in LDA and DMM.

  • Results

    The models significantly improve topic coherence, document clustering, and document classification, especially on datasets with few or short documents.

  • Takeaways & Limitations

    External latent feature information improves topic-to-word mappings and document-topic assignments relative to baseline models, particularly for small or short datasets.

  • Takeaways & Limitations

    The approach is slow on very large corpora, and the paper does not evaluate it on very large corpora.

Abstract

from arXiv · show

Probabilistic topic models are widely used to discover latent topics in document collections, while latent feature vector representations of words have been used to obtain high performance in many NLP tasks. In this paper, we extend two different Dirichlet multinomial topic models by incorporating latent feature vector representations of words trained on very large corpora to improve the word-topic mapping learnt on a smaller corpus. Experimental results show that by using information from the external corpora, our new models produce significant improvements on topic coherence, document clustering and document classification tasks, especially on datasets with few or short documents.

1 Introduction

Conventional topic models can learn weak word-topic distributions from small or short corpora, motivating models that combine corpus evidence with external latent feature representations. The paper proposes LF-LDA and LF-DMM, which integrate latent feature word vectors into Dirichlet multinomial topic models and improve several evaluations, especially for sparse datasets.

  • Small or short document collections provide limited co-occurrence evidence for estimating document-to-topic and topic-to-word distributions.
  • External knowledge has improved topic representations, but irrelevant topics in a larger corpus can consume the model’s topic space.
  • Latent feature vectors model topics in a high-dimensional space and have achieved strong performance across many NLP tasks.
  • The proposed approach supplements a multinomial topic model learned from a small corpus with latent feature representations trained on a large external corpus.
  • The paper proposes LF-LDA and LF-DMM by replacing each model’s topic-to-word Dirichlet multinomial component with a mixture of multinomial and latent feature components.
  • The new models significantly improve topic coherence, document clustering, and document classification, especially on corpora with few or short documents.

2 Background

The background introduces LDA and DMM as Dirichlet multinomial topic models, contrasts their document-topic assumptions, and describes latent feature vectors learned from contextual word prediction. These models address sparse evidence in short texts through different topic assignments and estimation strategies.

  • LDA model: LDA represents each document as a topic distribution and each topic as a word distribution over a fixed vocabulary.
  • LDA model: LDA generates words by assigning topic indicators and drawing each word from the selected topic’s categorical word distribution.
  • LDA model: LDA inference uses collapsed Gibbs sampling to sample each word’s topic assignment conditional on all other topic assignments.
  • LDA model: The count notation tracks topic-word occurrences, document-word totals, excluded assignments, and vocabulary size during inference.
  • DMM model for short texts: Short or few-document corpora are challenging because of data sparsity and limited textual context.
  • DMM model for short texts: DMM assumes one topic per document, selecting that topic before generating every document word from the same topic.
  • DMM model for short texts: Collapsed Gibbs sampling for DMM samples each document’s topic conditional on the assignments of all other documents.
  • Latent feature representations: Latent feature vectors are learned from co-occurrence counts or neural models that predict words from window-based context.

3 New latent feature topic models

The paper introduces LF-LDA and LF-DMM, combining Dirichlet multinomial topic models with latent feature word representations and Gibbs-based inference. The models use a mixture of corpus-estimated and externally trained components to improve topic-word modeling.

  • LF-LDA and LF-DMM combine latent feature models with LDA and DMM, respectively, and include Gibbs sampling procedures.The models extend two Dirichlet multinomial topic models with latent feature representations.
  • The combined models replace each original topic-to-word component with a mixture of a Dirichlet multinomial component and a latent feature component.This preserves the original topic-model structure while adding latent feature information.
  • Latent feature word probabilities use CatE log-space scores formed from topic and word latent-feature vectors, with pretrained word vectors fixed.The topic vector is learned while the word-vector matrix comes from external pretrained representations.
  • LF-LDA samples a topic and component indicator for each word, whereas LF-DMM assigns one shared topic to every word in a document.Both models select between the Dirichlet multinomial and latent feature components for word generation.
  • Inference integrates out selected variables and uses approximate Gibbs sampling, with MAP estimation used for topic vectors rather than posterior sampling.The LF-DMM approximation factorizes across words and is described as reasonably accurate for short documents.
  • Direct LF-DMM sampling is computationally expensive because topic and component-selection variables are coupled and yield 2^(N_wd) possible selections.The paper approximates the conditional distribution to simplify this coupling; topic vectors are optimized with L-BFGS under L2 regularization.

4 Experiments

The experiments compare the new LF-LDA and LF-DMM models with baseline LDA and DMM across topic coherence, document clustering, and document classification. They emphasize settings with limited topic-to-word evidence and examine the effect of mixture weighting and pretrained vector sets.

  • The evaluation compares LF-LDA and LF-DMM against baseline LDA and DMM on topic coherence, document clustering, and document classification.Topic coherence assesses topic-word associations, while clustering and classification assess document-topic assignments.
  • The experiments focus on corpora with few or short documents, where topic-to-word distributions have little available data.The study also investigates which λ values perform well and compares two pretrained word-vector sets.

4.1 Experimental setup

The experimental setup uses pretrained Google and Stanford word vectors, several cleaned news, newsgroup, and Twitter datasets, and repeated evaluations with specified sampling procedures. It includes subsets designed to test short-document and small-corpus conditions.

  • Pretrained word vectors: The study uses Google Word2Vec and Stanford GloVe vectors, both 300-dimensional representations trained on large external corpora.Google vectors cover 3 million words and phrases from 100 billion Google News words; Stanford vectors cover 2 million words from 42 billion Common Crawl tokens.
  • Model variants: The models are named w2v-LDA, glove-LDA, w2v-DMM, and glove-DMM according to the pretrained vector set and topic-model family.
  • Datasets: Experiments use 20-Newsgroups, TagMyNews, TagMyNews titles, and the Sanders Twitter corpus.The datasets contain documents or Tweets grouped into known categories for evaluation.
  • Preprocessing: Preprocessing lowercases text, removes non-alphabetic characters, stop words, short words, infrequent words, and words absent from both vector sets.The frequency thresholds differ between the 20-Newsgroups and TagMyNews datasets.
  • Preprocessing: The cleaned datasets removed 1,366, 27, and 12 words from 20-Newsgroups, TagMyNews, and TagMyNews titles, respectively.
  • Datasets: The N20short subset contains documents with fewer than 21 words, while N20small contains 400 documents sampled evenly across the 20 groups.These subsets target short-document and limited-corpus conditions.
  • Hyperparameters: The DMM concentration hyperparameter was set to 0.01, and α was set to 0.1 based on reported performance considerations.
  • Evaluation procedure: Baseline models ran for 2,000 iterations, while new models used 1,500 baseline iterations followed by 500 additional iterations after initialization.Results report means and standard deviations over ten repetitions.

4.2 Topic coherence evaluation

The models evaluate topic coherence through NPMI, using word co-occurrences from an external Wikipedia corpus. Latent-feature models improve coherence over baselines, with the latent-feature-only setting best on N20short.

  • Evaluation metric: NPMI measures semantic coherence among each topic’s top-15 words, averaged across topics.Probabilities come from a 10-word sliding window over English Wikipedia’s 4.6 million articles.
  • Evaluation metric: A higher NPMI score indicates more semantically coherent topic distributions.
  • Quantitative analysis: λ = 1.0 gives the highest NPMI on N20short with both 20 and 40 topics.At this setting, only the latent feature model is used to produce topic distributions.
  • Quantitative analysis: Latent-feature models produce significantly higher NPMI scores than baseline models across TMN, TMNtitle and Twitter.
  • Vector comparison: Google word2vec and Stanford GloVe vectors show different performance depending on dataset size, document length and topic count.Google vectors are better for several small or short datasets at low T, while the opposite pattern holds on full N20.
  • Qualitative analysis: w2v-DMM produces qualitatively purer and more clearly distinguishable topics than baseline DMM on TMNtitle.Examples include focused Japan-earthquake, Arab Spring, sport and politics topics.

4.3 Document clustering evaluation

Document clustering assigns each document its highest-probability topic and evaluates the resulting assignments with Purity and NMI. The latent-feature models generally improve clustering, especially for short or small datasets, although gains depend on corpus and topic count.

  • Evaluation setup: Purity and NMI evaluate clustering after assigning each document its highest-probability topic.NMI ranges from 0.0 to 1.0, with higher scores indicating better clustering.
  • Mixture-weight analysis: At λ = 1.0 with 20 topics, glove-LDA scores 1%+ higher than w2v-LDA on both Purity and NMI.With 40 topics, the two latent-feature models are equivalent and each improves both metrics by 2%+ absolute over baseline LDA.
  • Mixture-weight analysis: At λ = 0.6 with 40 topics, w2v-LDA improves over LDA by 4.4% on Purity and 4.3% on NMI.The improvement is reported as statistically significant with p < 0.01.
  • Baseline comparison: At T = 80 on TMNtitle, DMM achieves about 7+% higher Purity and NMI than LDA.
  • New models versus baselines: Across most tests, the new models outperform baselines, including 6.0% higher NMI on N20small and 6.1% and 2.5% higher Purity on TMN and TMNtitle.On Twitter, improvements at T = 4 are 3.9% in Purity and 5.3% in NMI.
  • Dataset and topic-count effects: For T ≤ 7 on large N20, TMN and TMNtitle datasets, new and baseline models obtain similar clustering results.At higher T, new models do better on short TMN and TMNtitle, while baseline LDA is slightly better on N20.
  • Vector comparison: Google vectors outperform Stanford vectors on small datasets, whereas Stanford vectors are better on large datasets at smaller T and converge at T = 80.Both vector sets produce similar results on Twitter.

4.4 Document classification evaluation

Document classification predicts ground-truth labels from each document’s topic-proportion vector using SVM. The new models generally improve F1, particularly on small and short datasets.

  • Evaluation setup: Classification uses SVM to predict ground-truth labels from each document’s topic-proportion vector.
  • Model comparison: On the TMN dataset, LDA performs better than DMM for classification.This differs from the clustering comparison reported for short datasets.
  • New models versus baselines: On most evaluations, the new models achieve higher F1 scores than the baseline models.
  • New models versus baselines: On N20small and Twitter, w2v-LDA obtains 5+% higher F1 than LDA when topic count equals the number of ground-truth labels.
  • New models versus baselines: w2v-DMM achieves 5.4% and 2.9% higher F1 than DMM on TMN and TMNtitle with T = 80, respectively.
  • Vector comparison: The comparison between Google and Stanford vectors for classification is similar to their comparison for clustering.

4.5 Discussion

The discussion distinguishes evaluation goals: external-corpus information is most useful for coherence, while clustering and classification require corpus-specific information. An intermediate mixture balances these sources.

  • Interpretation of λ: Topic coherence performs best at λ = 1 because its NPMI evaluation uses word co-occurrences from the external Wikipedia corpus.
  • Interpretation of λ: Clustering and classification depend more strongly on idiosyncratic properties of the smaller topic-modeling corpora.
  • Interpretation of λ: For clustering and classification, λ = 0.6 performs best by combining external latent-feature information with corpus-specific multinomial topic-word distributions.
  • Cross-domain behavior: Latent-feature vectors improve performance even when the topic-modeling corpus differs substantially from the vectors’ external training domain, as in Twitter experiments.
  • Vector comparison: Google and Stanford latent-feature vectors produce very similar results, with no clear reason to prefer either in these topic-modeling applications.

5 Conclusion and future work

The paper proposes LF-LDA and LF-DMM, integrating latent feature representations into LDA and DMM, and reports improvements across topic coherence, clustering, and classification. The authors also identify unresolved questions about the source of gains, sampling approximations, vocabulary coverage, and scalability.

  • Contributions: LF-LDA and LF-DMM integrate latent feature representations into LDA and DMM to improve topic models.
  • Results: Our models outperformed baseline models on topic coherence across all 6 experimental datasets.
  • Results: Document clustering and classification results improved document-topic assignments, especially for datasets with few or short documents.
  • Open questions: The mechanism behind the gains remains uncertain because latent feature vectors use smaller context windows than bag-of-words topic models.Preliminary experiments suggest context-window size alone does not explain the improvements.
  • Future work: The Gibbs sampler uses a simple approximation because Gamma-function ratios are not cheap to sample, motivating alternative sampling methods.Metropolis-Hastings is suggested as one possible alternative.
  • Future work: The approach excludes words absent from both vector sets, and the authors suggest learning vectors for unseen words and fine-tuning seen-word vectors.
  • Limitations: The approach has not been evaluated on very large corpora and is slow on them, although gains are greatest when corpora are small.Variational Bayesian inference is suggested as a possible efficiency solution.
Loading 1810.06306v1…