PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Neural Topic Models: Methods, Applications, and Challenges

Xiaobao Wu, Thong Nguyen, Anh Tuan Luu

arXiv:2401.15351v2cs.CLcs.AIcs.IR

TL;DR

Topic models need evaluation approaches that support comparison, while conventional models also face inference and scalability challenges. This survey organizes neural topic models by structure and scenario, reviews their applications and challenges, and reports that its semantic-aware diversity metric aligns more closely with human ratings.

  • Problem

    Conventional topic models require complex model-specific inference and typically nonparallel algorithms, while varying evaluation metrics and settings impede comparison across studies.

    Source: §1 · p. 2, §6.1.1 · p. 14

  • Method

    The survey organizes NTM methods by neural structure and scenario, discusses applications and challenges, and introduces a topic-diversity metric that accounts for word semantics.

    Source: §1 · p. 3, §7.2.1 · p. 15

  • Results

    TSD achieves relatively higher correlation scores with human ratings across the reported datasets and average correlation.

    Source: §7.2.2 · p. 16

  • Takeaways & Limitations

    The authors intend the survey as a resource for NTM researchers and as a contribution to advancing NTM research.

    Source: §9 · p. 17

    Inconsistent metric use and settings confine evaluation results to specific studies, impeding comparison across papers.

    Source: §6.1.1 · p. 14

Abstract

from arXiv · show

Topic models have been prevalent for decades to discover latent topics and infer topic proportions of documents in an unsupervised fashion. They have been widely used in various applications like text analysis and context recommendation. Recently, the rise of neural networks has facilitated the emergence of a new research field -- Neural Topic Models (NTMs). Different from conventional topic models, NTMs directly optimize parameters without requiring model-specific derivations. This endows NTMs with better scalability and flexibility, resulting in significant research attention and plentiful new methods and applications. In this paper, we present a comprehensive survey on neural topic models concerning methods, applications, and challenges. Specifically, we systematically organize current NTM methods according to their network structures and introduce the NTMs for various scenarios like short texts and bilingual documents. We also discuss a wide range of popular applications built on NTMs. Finally, we highlight the challenges confronted by NTMs to inspire future research. We accompany this survey with a repository for easier access to the mentioned paper resources: https://github.com/bobxwu/Paper-Neural-Topic-Models.

1 Introduction

Conventional topic models face labor-intensive inference and scalability limits, motivating neural alternatives. This survey organizes NTM methods, scenarios, applications, and challenges, and proposes a semantic-aware topic-diversity metric.

1 Introduction

  • Conventional topic models require complex, model-specific parameter inference, and their typically nonparallel algorithms can make training slow.Training a probabilistic dynamic topic model on 10k documents takes two days in the cited example.Source: §1 · pp. 1–2
  • NTMs use neural networks and gradient back-propagation to infer parameters, with flexibility for diverse scenarios and parallel computing for large datasets.The survey identifies these properties as advantages over conventional approaches.Source: §1 · p. 2
  • The survey organizes NTM methods by network structure and covers a wider range of scenarios, including short-text, cross-lingual, and dynamic topic modeling.It also provides background and illustrations for the scenarios.Source: §1 · p. 3
  • The survey discusses NTM applications for tasks such as text analysis and generation.It also identifies challenges to motivate future research.Source: §1 · p. 3
  • The authors propose topic semantic-aware diversity (TSD), which measures diversity using word semantics and is reported to align more closely with human judgment.The survey presents TSD as part of its discussion of NTM challenges.Source: §1 · p. 3

2 Preliminary of Topic Models

This section introduces topic-modeling preliminaries and then presents the basic and popular NTM framed as a Variational AutoEncoder (VAE).

2 Preliminary of Topic Models

  • The section introduces topic-modeling preliminaries, including the problem setting, notations, and evaluation, before presenting a basic NTM in the VAE framework.Source: §2 · p. 3

2.1 Problem Setting and Notations

The section defines topic modeling over a collection of documents with a manually selected topic count. It describes topics as vocabulary distributions and document-topic distributions as each document’s topic proportions.

2.1 Problem Setting and Notations

  • The setup considers N documents with V unique words and uses K, usually selected manually for the dataset and task, as the number of latent topics.Source: §2.1 · pp. 3–4
  • Each latent topic is represented by a topic-word distribution over the vocabulary.Source: §2.1 · p. 4
  • A document’s doc-topic distribution θ indicates which topics it contains.Source: §2.1 · p. 4

2.2 Evaluation of Topic Models

Topic-model evaluation is unsettled because ground-truth labels are absent, and no single measure captures all aspects of model quality. The survey reviews predictive, topic-quality, downstream-task, and visualization approaches.

2.2.1 Perplexity

  • Perplexity measures prediction of held-out documents, but can conflict with human judgments, vary across models’ likelihood calculations, and miss content-analysis utility.The survey attributes these issues to differing sampling, approximation, and distribution-normalization choices across models.Source: §2.2.1 · p. 4

2.2.2 Topic Coherence

  • Topic coherence assesses relationships among a topic’s top words; NPMI averages normalized mutual information across word pairs, with large external reference corpora recommended for fairer comparisons.CV compares cosine similarity between NPMI score vectors and is reported as more consistent with human judgment than NPMI, UCI, and UMass.Source: §2.2.2 · pp. 4–5

2.2.3 Topic Diversity

  • TU rewards rare top words, TR measures their recurrence in other topics, and TD counts unique top words; these metrics treat word uniqueness as diversity.Their shared-word assumption is problematic when one word belongs to distinct semantic topics, such as “fruit” and “company” for “apple”; the paper proposes a metric to address this.Source: §2.2.3 · p. 5
  • Topic diversity remains unresolved when distinct topics share words, as with “apple” in “fruit” and “company” topics; the paper proposes a metric to address this issue.Source: §2.2.3 · p. 5

2.2.4 Downstream Task Performance

  • Downstream evaluation uses document-topic distributions as features for classification or clustering, assessed with accuracy or F1 for classification and Purity or NMI for clustering.Clustering can assign each document its largest-proportion topic or apply algorithms such as K-Means or DBSCAN to the distributions.Source: §2.2.4 · pp. 5–6

2.2.5 Visualization

  • Visualization commonly displays topic top words and document-topic distributions, or places documents on a 2D canvas after dimensionality reduction.Examples include pyLDAvis, word clouds, and t-SNE.Source: §2.2.5 · p. 6

2.3 Basic NTM based on VAE

The basic neural topic model uses a VAE to infer document-topic distributions and reconstruct documents. Its objective combines reconstruction error with KL-divergence regularization.

2.3 Basic NTM based on VAE

  • A VAE-based NTM uses an encoder to infer document-topic distributions from documents, with a logistic-normal latent prior approximating a symmetric Dirichlet prior.The prior uses mean vector μ0 and diagonal covariance matrix Σ0; the variational distribution is parameterized by Θ.Source: §2.3 · p. 6
  • The encoder uses MLPs on document bag-of-words inputs and samples the latent variable with the reparameterization trick before restricting the result to the simplex.Source: §2.3 · p. 6
  • A decoder maps document-topic distributions to word-generation probabilities and samples documents from a multinomial; training combines reconstruction error and KL divergence.The KL term regularizes the variational distribution against the prior.Source: §2.3 · p. 6
  • This VAE structure is the foundation for later NTMs with different structures intended to improve performance or address varied application scenarios.Source: §2.3 · p. 6

3 NTMs with Different Structures

The survey organizes neural topic models by network structure, covering variations in priors and representations as well as models that incorporate metadata, graphs, and learning objectives. It also reviews clustering and semantic-relation approaches alongside other architectures.

3.1 NTMs with Various Priors

  • Beyond Gaussian priors, NTMs approximate or use Gaussian-softmax, stick-breaking, Weibull, auxiliary-uniform, and rounded-posterior distributions, including approximations to Dirichlet priors.The survey notes Gaussian priors are common because reparameterization and analytical KL-divergence calculations are straightforward.Source: §3.1 · pp. 6–7

3.2 NTMs with Embeddings

  • Embedding-based models represent topics with word and topic embeddings; variants use pretrained vectors, transport-based reconstruction, hyperbolic spaces, or clustering regularization to reduce topic repetition.ECRTM’s clustering regularization treats topic embeddings as centers and word embeddings as samples, using optimal transport to avoid topic collapse.Source: §3.2 · p. 7

3.3 NTMs with Metadata; 3.4 NTMs with Graph Neural Networks

  • Metadata-aware models condition document generation on labels or covariates, while graph-based models represent documents with biterm, document-word, co-occurrence, or semantic graphs.Other variants jointly model documents with labels, images, or document networks; graph models also include microblog and commonsense-knowledge settings.Source: §3.3 · p. 7, §3.4 · pp. 7–8

3.5 NTMs with Adversarial Training

  • Adversarial NTMs use generated documents and discriminators; bidirectional adversarial training adds doc-topic inference, and cycle-consistency constraints extend the approach.The initial adversarial model cannot infer doc-topic distributions because it maps documents directly to TF-IDF representations.Source: §3.5 · p. 8

3.6 NTMs with Pre-trained Language Models

  • Pretrained-language-model variants provide contextual features, distill BERT predictions into pseudo-BoW, or use multilingual BERT, though the latter do not discover aligned cross-lingual topics.Some models reconstruct both real and pseudo bag-of-words representations.Source: §3.6 · p. 8

3.7 NTMs with Contrastive Learning; 3.8 NTMs with Reinforcement Learning

  • Contrastive-learning NTMs construct positive and negative pairs from words, topic semantics, or quantized doc-topic distributions, and also apply contrastive objectives to embeddings and topics.Reinforcement-learning variants use topic coherence as a reward during training; later work also tracks diversity and coherence.Source: §3.7 · p. 8, §3.8 · p. 8

3.9 Other NTMs

  • Other architectures include autoregressive DocNADE, sparsemax and Wasserstein-autoencoder models, discrete topic assignments, and interventions aimed at long-tailed bias.DocNADE predicts words conditioned on previous words and uses document hidden states to infer topic proportions; n-gram models also learn from positive and sampled negative documents.Source: §3.9 · pp. 8–9

3.10 Topic Discovery by Clustering; 3.11 NTM with Semantic-Relations

  • Clustering-based topic discovery is computationally efficient but often cannot infer doc-topic distributions; FASTopic instead uses optimal transport between document, topic, and word embeddings to model distributions and semantic relations.BERTopic estimates doc-topic distributions from within-document term weights; FASTopic is presented as an alternative to VAE and simple clustering structures.Source: §3.10 · p. 9, §3.11 · p. 10

4 NTMs for Various Scenarios

The survey reviews NTMs tailored to distinct settings, including hierarchies, sparse short texts, multiple languages, temporal evolution, topic correlations, and continual learning.

4.1 Hierarchical NTMs

  • Hierarchical NTMs represent topics at multiple semantic granularities; non-parametric structures can grow dynamically, whereas parametric structures require preset hierarchy dimensions.The survey says parametric settings fit better when enough prior knowledge is available to determine the hierarchy, while non-parametric approaches still require tuning growth-related hyperparameters.Source: §4.1 · p. 10

4.2 Short Text NTMs

  • Short-text NTMs address sparse word co-occurrence using topic-distribution regularization or quantization, negative sampling, graphs, entity vectors, and contrastive learning.NQTM’s carefully initialized quantization vectors produce sharper doc-topic distributions; data augmentation is also used to mitigate sparsity.Source: §4.2 · p. 11

4.3 Cross-lingual NTMs

  • Cross-lingual NTMs seek aligned topics across languages; methods transform topic-word distributions or use mutual information, while multilingual BERT approaches may infer document-topic distributions without aligned topics.Aligned topics can support cross-lingual text analysis without supervision.Source: §4.3 · pp. 11–12

4.4 Dynamic NTMs

  • Dynamic NTMs model topic evolution across time slices, with approaches chaining topic embeddings, learning temporal priors, or using self-attention and contrastive learning.Some methods instead model topic activity while leaving topics invariant, which the survey says does not precisely follow the original definition of dynamic topic modeling.Source: §4.4 · p. 12

4.5 Correlated NTMs

  • Correlated NTMs model relationships among latent topics; a VAE-based approach uses a centralized transformation flow and a lower bound to regulate its KL-divergence term.Source: §4.5 · p. 12

4.6 Lifelong NTMs

  • Lifelong NTMs accumulate topical knowledge from document streams to address data sparsity while retaining prior knowledge and limiting catastrophic forgetting.Reported mechanisms include topic regularization across domains and selective replay of relevant historical documents.Source: §4.6 · pp. 12–13

5 Applications of NTMs

The survey describes NTMs applied to text analysis, text generation, and content recommendation.

5.1 Text Analysis

  • Text-analysis applications include scientific-article analysis, short-text classification, disinformation classification, discourse analysis, epidemic tracking, sentiment analysis, and topic mining.Some systems combine topic models with citation graphs or classifiers; others use dynamic or short-text NTMs for domain-specific analysis.Source: §5.1 · p. 13

5.2 Text Generation

  • Text-generation applications use topic models to represent semantics, address information sparsity in long stories, control summarization semantics, or guide keyphrase generation.One long-story approach samples interrelated words from a document-topic distribution to form a skeleton for a Transformer.Source: §5.2 · p. 13

5.3 Content Recommendation

  • For recommendation, NTMs have been combined with review recommenders and graph models for citation recommendation.Source: §5.3 · p. 13

6 Challenges of NTMs

The survey identifies reliable evaluation, topic quality, and hyperparameter sensitivity as key challenges for neural topic models.

6.1 Lacking Reliable Evaluation; 6.1.1 Absence of Standard Evaluation Metrics

  • Evaluation remains unreliable because human judgment is costly, automatic metrics vary across studies, and coherence or diversity scores may disagree with human assessments.Metric and setting inconsistencies limit cross-paper comparability; the survey notes that some benchmarks find conventional LDA can still outperform NTMs in certain aspects.Source: §6.1.1 · pp. 13–14, §6.1 · p. 13

6.1.2 Lacking Standardized Dataset Pre-processing Settings

  • Different preprocessing settings can substantially change results, raising questions about whether reported improvements generalize across datasets and settings.The survey calls standardized preprocessing an imperative prerequisite for reliable and consistent comparisons.Source: §6.1.2 · p. 14

6.2 Low-Quality Topics

  • Discovered topics may be trivial, repetitive, or both, weakening interpretability and usefulness for downstream tasks.Trivial topics contain uninformative words; repetitive topics share words and can leave document semantics undiscovered.Source: §6.2 · pp. 14–15

6.3 Sensitivity to Hyperparameters

  • NTMs’ many consequential hyperparameters, and poor performance by some models with many topics, require careful tuning and can curtail generalization.Examples of influential settings include dropout probability, batch size, and learning rate.Source: §6.3 · p. 15

7 Topic Semantic-aware Diversity

The section introduces Topic Semantic-aware Diversity (TSD), a metric that incorporates word semantics into topic diversity. It argues that word-pair uniqueness better handles shared words with different meanings and reports closer alignment with human ratings.

7.1 Problem of Previous Diversity Metrics

  • Uniqueness-based metrics can undervalue diversity when topics share polysemous words: authors judge fruit- and technology-related, and Steve Jobs- and employment-related topics most diverse, despite TU 0.867 and TD 0.733.Source: §7.1 · p. 15

7.2 Topic Semantic-aware Diversity

7.2.1 Definition of Topic Semantic-aware Diversity

  • TSD measures uniqueness of unordered word pairs among topic top words, using pair frequency rather than the uniqueness of individual words.A pair contributes when it occurs once across the top words of all K topics; the authors note TSD degrades to TD when counting individual words.Source: §7.2.1 · p. 15
  • When shared words pair differently across topics, TSD can treat the topics as distinct; repeated pairs lower its score, and identical topics yield zero for both TD and TSD.The three cases are presented as comparisons among TSD, TU, and TD.Source: §7.2.1 · pp. 15–16, Table 2 · p. 16

7.2.2 Evaluation Results

  • The evaluation compares metric correlations with human ratings after LDA and NSTM generate topics from real-world datasets.The datasets include NeurIPS, ACL, NYT, and Wikitext103.Source: §7.2.2 · p. 16
  • TSD achieves relatively higher correlations with human ratings across datasets, supporting closer alignment with human judgments of topic diversity.The reported comparison includes correlations for different datasets and their average.Source: §7.2.2 · p. 16, Table 3 · p. 16

8 Topic Model Toolkits

This section surveys toolkits for topic modeling, noting both implementation gaps and newer platforms that support broader NTM research and application.

8 Topic Model Toolkits

  • Earlier toolkits often omit NTM implementations, dataset preprocessing, or evaluations, limiting their coverage of practical requirements.Source: §8 · pp. 16–17
  • OCTIS includes several NTM methods, evaluations, and Bayesian parameter optimization, while TopMost covers more recently released NTMs and more scenarios.Source: §8 · p. 17
  • TopMost separates model implementations from training to ease extension, and the toolkits provide a foundation for exploring models and using them in applications.Source: §8 · p. 17

9 Conclusion

The conclusion presents NTMs as a flexible and scalable development in topic modeling and summarizes the survey’s coverage of methods, applications, and research challenges.

9 Conclusion

  • NTMs have attracted attention for flexibility and scalability, including avoiding model-specific derivations and handling large-scale datasets efficiently.Source: §9 · p. 17
  • The survey reviews NTM network structures, their applicability across scenarios, and popular applications built on them.Source: §9 · p. 17
  • The authors identify and discuss challenges for future NTM research, aiming for the survey to support researchers interested in the field.Source: §9 · p. 17
Loading 2401.15351v2…