Source-linked AI summary
Discovering Discrete Latent Topics with Neural Variational Inference
Yishu Miao, Edward Grefenstette, Phil Blunsom
TL;DR
As topic models grow more expressive, inference becomes increasingly difficult to derive and perform efficiently. This paper introduces neural topic distributions trained with variational inference, including a recurrent stick-breaking model for notionally unbounded topics. Across standard corpora, the neural models achieve state-of-the-art performance.
Problem
As topic models become more expressive, traditional inference methods become increasingly complex, making fast and accurate inference difficult, particularly for non-conjugate models.
Method
The paper parameterises topic distributions with neural networks trained by variational inference, using Gaussian Softmax, Gaussian Stick Breaking, and Recurrent Stick Breaking constructions.
Results
The neural models achieve state-of-the-art performance across standard document corpora, with GSM, GSB, and RSB significantly outperforming benchmark LDA and NVLDA models on perplexity.
Takeaways & Limitations
Neural topic models combine backpropagation-based training with interpretable probabilistic topic representations and can dynamically increase the number of active topics.
Abstract
from arXiv · showhide
Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.
1. Introduction
As topic models become more expressive, inference grows more complex, motivating neural variational approaches that combine neural parameterisation with probabilistic topic modelling. The paper proposes neural topic models and evaluates them against neural document models and traditional probabilistic topic models.
- More expressive topic models make fast and accurate inference increasingly difficult, especially for non-conjugate models.
- Neural variational inference parameterises a generative model’s posterior with a neural network and jointly trains both components by backpropagation.
- The paper introduces Gaussian Softmax, Gaussian Stick Breaking, and Recurrent Stick Breaking topic distributions conditioned on multivariate Gaussian draws.
- The Recurrent Stick Breaking process uses an RNN to progressively break the stick, yielding a neural analogue of a Dirichlet Process topic model.
- Experiments compare the proposed models with neural document models and traditional probabilistic topic models across multiple datasets.
2. Parameterising Topic Distributions
The paper replaces conventional topic-distribution parameterisation with neural transformations of Gaussian latent variables. Gaussian Softmax and Gaussian Stick Breaking model finite topic distributions, while Recurrent Stick Breaking extends the construction to dynamically generate an unbounded sequence of topics.
- Traditional Dirichlet priors enable closed-form mean-field variational updates for latent topic proportions and word assignments.
- The proposed models use neural networks to parameterise multinomial topic distributions instead of relying on the traditional formulation.
- Neural variational inference uses an MLP-conditioned Gaussian q(θ|d) to approximate p(θ|d), with reparameterisation providing an unbiased, low-variance gradient estimator.
- Gaussian Softmax Construction: Gaussian Softmax passes a Gaussian random vector through softmax to parameterise document topic probabilities.
- Gaussian Stick Breaking Construction: Gaussian Stick Breaking transforms Gaussian latent variables into sigmoid break proportions and deterministically converts them into topic probabilities whose sum is one.
- Gaussian Stick Breaking Construction: The stick-breaking construction introduces a non-parametric aspect but breaks exchangeability relative to the Dirichlet-process construction.
- Recurrent Stick Breaking Construction: Recurrent Stick Breaking uses an RNN to generate successive binomial logits and can dynamically produce new breaks for an unbounded number of topics.
3. Models
The models parameterize topic distributions with neural networks and train them through neural variational inference. Recurrent stick breaking further supports dynamically expanding topic sets without fixed truncation.
- Neural topic models: Topic-word distributions β are formed from semantic similarities between learned topic vectors and word vectors.The resulting β consists of K word-distribution simplexes for finite models.
- Neural variational inference: An inference network maps each document d to variational parameters µ(d) and σ(d), then samples θ with the Gaussian reparameterization trick.This provides an unbiased, low-variance gradient estimator for joint backpropagation.
- Neural variational inference: The sampled topic proportions allow topic assignments z to be integrated out, eliminating a separate variational approximation for z.The resulting lower bound can be optimized by gradients of both generative and variational parameters.
- Recurrent neural topic models: RSB dynamically generates topic vectors and topic proportions, enabling a notionally unbounded number of topics without truncating variational inference.RNNTopic produces new topics as RNNSB performs additional stick breaks.
- Recurrent neural topic models: A likelihood-increase threshold γ determines whether a proposed topic becomes active, controlling the rate at which new topics are generated.The decision is computed over mini-batches during training.
- Topic versus document models: The same neural topic models can also be used as document models by removing topic-word assignment modeling and applying an implicit decoder.This produces a model variant related to NVDM, where words are generated from an unnormalized document representation.
4. Related Work
The work extends neural variational inference for topic models while differing from closely related approaches in how distributions and gradients are parameterized. Its experiments compare finite and unbounded neural models with established topic-model baselines.
- Existing topic-model extensions: Prior topic-model extensions address topic correlations, temporal dependencies, unbounded topic counts, and contextual information such as time, authorship, and labels.These extensions broaden the modeling settings beyond standard LDA.
- Experimental comparisons: The experiments compare GSM, GSB, and RSB with onlineLDA and NVLDA, while unbounded RSB-TF is compared with online HDP.Finite models use 50 or 200 topics across MXM, 20NewsGroups, and RCV1.
- Neural variational topic models: Unlike Srivastava and Sutton’s Laplace approximation around a Dirichlet-Multinomial model, these models directly parameterize multinomials with neural networks.The proposed model and variational parameters are jointly learned during inference.
5. Experiments
Experiments evaluate neural topic and document models on three corpora using perplexity and topic coherence. The results show strong performance, stable stick-breaking behavior, and dynamically learned topic counts.
- Experimental setup: The evaluation uses MXM song lyrics, 20NewsGroups, and Reuters RCV1-v2, with datasets ranging from 27,143 to 794,414 test or training cases.MXM has 210,519 training and 27,143 testing datapoints; 20NewsGroups has 11,314 training and 7,531 testing documents; RCV1-v2 has 794,414 training and 10,000 test cases.
- Perplexity: Among finite topic models, GSM achieves the lowest perplexity in most cases, while GSM, GSB, and RSB significantly outperform LDA and NVLDA.
- Perplexity: The truncation-free RSB model performs significantly better than the traditional HDP model on perplexity.
- Document models: Switching GSM, GSB, and RSB to implicit topic distributions improves generalisation and yields significantly better performance than NVDM and ProdLDA.On 20NewsGroups, RSB outperforms GSM and GSB in both the 50- and 200-topic settings, possibly because its recurrent sequence induces dependencies and helps escape local minima.
- Topic scalability: GSB and RSB remain stable when the maximum topic count far exceeds the dataset’s needs, whereas GSM performance worsens beyond 400 topics.RSB is better than GSB below 200 topics but slightly worse above 400, possibly because long RNN sequences are harder to learn.
- Topic scalability: RSB-TF dynamically increases active topics during training, typically stabilising between 200 and 300 on 20NewsGroups regardless of whether initialization starts at 10, 30, or 50 topics.Training perplexity decreases as active topics increase, and the model is not sensitive to its initial active-topic count.
6. Conclusion
The paper introduces neural topic models based on Gaussian softmax, Gaussian stick-breaking, and recurrent stick-breaking constructions. These models achieve state-of-the-art performance while supporting sparse and dynamically growing topic distributions.
- The proposed family parameterises document-level latent multinomial topic distributions with Gaussian Softmax, Gaussian Stick-Breaking, and Recurrent Stick-Breaking constructions.
- Stick-breaking produces sparse topic distributions, while recurrent networks allow a truncation-free variational method whose number of topics dynamically increases.
- The evaluation reports state-of-the-art performance across a range of standard document corpora.
A. Discovered Topics
The paper presents topics learned by neural topic models on 20NewsGroups using their highest-probability words. The comparison is organized around topic outputs from different neural models.
- Table 4 presents the top-10 words defining topics learned by different neural topic models on 20NewsGroups.
- The displayed comparison includes topics learned by the RSB model.
B. Topic Diversity
Topic diversity regularisation is applied to separate topic vectors in semantic space. It reduces redundancy, although it usually produces little perplexity improvement.
- The method regularises pairwise angles between topic vectors to diversify topics in the semantic space.It encourages a larger mean angle and suppresses angle variance so topics are pushed away from one another.
- The diversity regularisation coefficient λ is set to 0.1 in the experiments.
- Topic diversity regularisation reduces topic redundancy but does not significantly improve perplexity, with gains of 2∼5 in most cases.