Source-linked AI summary
Topic Modelling Meets Deep Neural Networks: A Survey
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, Wray Buntine
TL;DR
Conventional topic models face difficulties with inference complexity, large-scale computation, neural-network integration, and inconsistent evaluation. This paper surveys neural topic models, organizing their frameworks, applications, and open challenges. It emphasizes the breadth of NTM approaches and the need for better evaluation.
Problem
Conventional BPTMs face inference, scalability, GPU-integration, and joint-training limitations, while NTM evaluation remains difficult and inconsistent.
Method
The paper provides a focused, comprehensive review and taxonomy of neural topic models across frameworks, distributions, applications, and challenges.
Results
The survey identifies more than a hundred NTM models and variants spanning neural topic modelling frameworks and applications.
Takeaways & Limitations
NTMs constitute a flexible research area with applications including text generation, summarisation, translation, and short-text modelling.
Takeaways & Limitations
Comparisons across NTMs are limited because evaluation lacks a unified metric system and predictive-probability calculations are inconsistent.
Abstract
from arXiv · showhide
Topic modelling has been a successful technique for text analysis for almost twenty years. When topic modelling met deep neural networks, there emerged a new and increasingly popular research area, neural topic models, with over a hundred models developed and a wide range of applications in neural language understanding such as text generation, summarisation and language models. There is a need to summarise research developments and discuss open problems and future directions. In this paper, we provide a focused yet comprehensive overview of neural topic models for interested researchers in the AI community, so as to facilitate them to navigate and innovate in this fast-growing research area. To the best of our knowledge, ours is the first review focusing on this specific topic.
1 Introduction
Neural topic models emerged to address scalability, inference, and integration limitations of conventional Bayesian probabilistic topic models. This survey organizes the rapidly growing NTM literature, reviews applications, and identifies open problems.
- Conventional BPTMs require model-specific inference procedures whose complexity can grow with model complexity, while their design is difficult to automate.
- BPTMs can be difficult to scale efficiently on large collections, exploit GPUs, and integrate with other deep neural networks for joint training.
- NTMs leverage DNNs and deep generative models to improve topic-modelling performance, efficiency, and usability.
- More than a hundred NTM models and variants have been developed, including applications in text generation, document summarisation, and translation.
- The paper provides a taxonomy, reviews research developments and applications, and discusses remaining challenges and future directions.
2 Background, Definition, and Evaluation
Topic models infer latent topics and document-topic representations from corpus data, commonly reconstructing documents through learned projections. Their evaluation remains difficult because predictive, coherence, diversity, and downstream metrics capture different properties.
- Background and Definition: Documents can be represented as word sequences or bag-of-words vectors that record vocabulary-token occurrences.
- Background and Definition: A topic is an interpretable semantic concept represented as a distribution over vocabulary tokens, while a document has a distribution over shared topics.
- Architectures and Learning: Topic models learn document-to-topic and topic-to-document projections, with neural topic models typically parameterizing these processes using deep neural networks.
- Evaluation: Evaluation commonly uses predictive accuracy, topic coherence, topic diversity, and downstream application performance.
- Evaluation: Predictive accuracy is an incomplete evaluation signal because it may not reflect topic interpretability, may disagree with human judgments, and is inconsistently estimated across NTM studies.
3 Neural Topic Models with Amortised Variational Inference
VAE-NTMs combine neural encoders and decoders with amortised variational inference, adapting generative topic modelling to sparse counts, sequences, and interpretability requirements. The survey covers distributional variants, structured topics, metadata, short texts, and contextual or sequential inputs.
- VAE-NTMs: VAE-NTMs model topic inference and document generation with DNN-based encoders and decoders while optimizing an ELBO objective.The objective combines expected reconstruction likelihood with a KL-divergence regularizer.
- VAE-NTMs: Designing VAE-NTMs requires handling high-dimensional sparse count vectors or variable-length sequences while preserving interpretable topic representations.
- Variants of Distributions: VAE-NTM variants modify priors, data distributions, posterior approximations, decoder and encoder architectures, and topic-word distributions.
- Variants of Distributions: Topic-word matrices are often factorized into topic and word embeddings, representing topic-word relevance through distances in an embedding space.
- Structured and Conditional Models: Other extensions capture topic correlations or structures, incorporate metadata, and use specialized architectures for reviews and supervised tasks.
- NTMs for Short Texts: Short-text NTMs address insufficient word co-occurrence information using memory networks, graph neural networks, regularization, and reinforced content.
- Sequential and Contextual Models: Sequential and contextual NTMs use word order, sentence structure, topic structure, or contextual embeddings beyond bag-of-words inputs.
4 NTMs based on Other Frameworks
Beyond VAE-NTMs, neural topic models use autoregressive, adversarial, graph-based, and other frameworks. These approaches vary in their input representations, learning objectives, and ability to infer document-level topic distributions.
- Autoregressive Frameworks: Autoregressive NTMs condition word prediction on preceding words, with hidden units interpretable as topics and document hidden states representing topic distributions.
- Adversarial Frameworks: GAN-based NTMs generate word distributions from sampled topic distributions and use discriminators to distinguish generated from real document distributions.
- Adversarial Frameworks: The initial GAN-based model discovered interpretable topics but could not learn topic distributions for individual documents, motivating an encoder extension.
- Graph Frameworks: Graph-based NTMs represent corpora as biterm graphs or document-word bipartite graphs to apply graph neural networks for topic discovery.
- Other Frameworks: Other NTMs use n-gram prediction, competitive autoencoders, matrix factorization, reinforcement learning, Wasserstein autoencoders, and optimal transport.
5 Applications of NTMs
NTMs offer flexible representations and gradient-based inference, making them easier to integrate with broader neural models. Their global semantic representations complement sequence models in language understanding applications.
- NTMs represent document-topic and topic-word distributions as probability vectors or embeddings, enabling flexible integration into broader models.
- Gradient-based optimisation makes NTM inference more convenient to integrate with other DNNs for joint training.
- NTMs learn global semantic representations from bag-of-words data, complementing RNNs, transformers, and attention models that may miss long-range dependencies.
- NTMs have been applied to language models, text generation, and document summarisation.
6 Discussion
The survey identifies evaluation as a central challenge for NTMs because inconsistent metrics, settings, architectures, and datasets hinder comparisons. It argues that comprehensive evaluation is preferable to relying on a single metric.
- NTM comparisons are difficult because evaluation metrics and procedures vary across frameworks, architectures, and datasets.
- VAE-NTM perplexity based on ELBO cannot be directly compared with perplexity from models without ELBO.
- Topic coherence and downstream performance evaluations also vary in metrics and settings across papers.
- Evaluating topic models with only one metric, such as topic coherence, can produce a tendentious assessment.