Source-linked AI summary
Topic Modeling with Wasserstein Autoencoders
Feng Nan, Ran Ding, Ramesh Nallapati, Bing Xiang
TL;DR
The paper addresses limitations of VAE-based neural topic models by proposing W-LDA, which directly matches latent document-topic vectors to a Dirichlet prior within the WAE framework. It reports better topic quality than existing models, while finding advantages for MMD over GAN in high-dimensional Dirichlet matching and for encoder noise in improving coherence.
Problem
VAE-based topic models face posterior collapse, while topic modeling lacks a measure of topic diversity alongside coherence.
Method
W-LDA uses a Wasserstein autoencoder with direct Dirichlet-prior matching, MMD distribution matching, encoder-output noise, and Topic Uniqueness alongside NPMI.
Results
W-LDA produces significantly better topic quality than existing topic models in coherence and uniqueness, while MMD outperforms GAN for high-dimensional Dirichlet matching.
Takeaways & Limitations
Combining NPMI with Topic Uniqueness provides a more holistic evaluation of topic quality, and encoder noise significantly improves coherence without harming diversity.
Takeaways & Limitations
The authors were unable to train the GAN-based W-LDA because of vanishing gradients and identify improved GAN-based training as future work.
Abstract
from arXiv · showhide
We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.
1 Introduction
The paper proposes W-LDA, a Wasserstein autoencoder topic model that directly matches latent document-topic vectors to a Dirichlet prior and evaluates topics by both coherence and uniqueness.
- 1 Introduction: W-LDA directly enforces a Dirichlet prior on latent document-topic vectors through distribution matching, avoiding Gaussian approximation and posterior collapse.The model matches the aggregated posterior to the prior, helping latent codes remain distinct and supporting reconstruction.
- 1 Introduction: The proposed Topic Uniqueness measure complements NPMI by evaluating whether discovered topics are diverse rather than repetitive.The paper identifies topic diversity as an under-measured aspect of topic quality.
- 1 Introduction: W-LDA produces significantly better topic quality than existing topic models in both coherence and uniqueness.
- 1 Introduction: MMD with a carefully chosen kernel has a key performance advantage over GAN for matching high-dimensional Dirichlet distributions.
- 1 Introduction: Adding noise to the encoder output significantly boosts topic coherence and can potentially generalize to WAE models.
2 Related Work
The work builds on adversarial and Wasserstein autoencoders, positioning W-LDA as a topic model that performs distribution matching in the latent document-topic space.
- 2 Related Work: WAE extends adversarial autoencoder ideas by permitting MMD as an alternative to adversarial training for matching aggregated posteriors to priors.
- 2 Related Work: W-LDA is presented as the first topic model based on the WAE framework and differs from ATM by matching distributions in latent document-topic space.ATM performs distribution matching in the higher-dimensional vocabulary space and evaluates only topic coherence in the cited discussion.
- 2 Related Work: The paper relates topic disentanglement to coherent and distinct topics, drawing on prior evidence that WAE can learn more disentangled representations than VAE.
3 Background
The background introduces LDA's latent document-topic representation and inference problem, then explains how WAEs replace VAE-style posterior regularization with aggregated-posterior matching using divergences such as GAN or MMD.
- 3.1 Latent Dirichlet Allocation: LDA represents each document with a mixed-membership topic vector drawn from a Dirichlet prior, then generates words through topic and word distributions.The number of topics is fixed, and each topic is a probability distribution over the vocabulary.
- 3.1 Latent Dirichlet Allocation: Inference in LDA determines the conditional distribution of the document-topic vector given an observed document.
- 3.2 Wasserstein Auto-encoder: A WAE generates target-domain examples by sampling a latent code from a prior and passing it through a decoder, while optimizing reconstruction and distribution-matching terms.The WAE objective is justified through optimal-transport minimization.
- 3.2 Wasserstein Auto-encoder: Unlike VAE regularization on each posterior, WAE regularizes the aggregated posterior, the encoded distribution obtained by averaging conditional encoders over the data.
- 3.2 Wasserstein Auto-encoder: WAE distribution matching can use either GAN-based divergence or MMD between the aggregated posterior and the prior.MMD compares kernel-induced mean embeddings in a reproducing kernel Hilbert space.
4 W-LDA
W-LDA maps bag-of-words documents to simplex-valued topic vectors and reconstructs them through a neural decoder. It matches the aggregated latent distribution with MMD using a simplex-sensitive kernel, balances reconstruction and regularization, and mixes Dirichlet noise into encoder outputs during training.
- 4.1 Encoder-decoder: W-LDA maps bag-of-words documents to K-dimensional topic vectors with an MLP encoder and reconstructs word distributions with a single-layer decoder.The encoder applies softmax to produce θ, while the decoder applies softmax to produce the reconstructed distribution ˆw.
- 4.2 Distribution matching: MMD-based distribution matching is preferred because the GAN variant suffers vanishing gradients, while a simplex-geodesic information diffusion kernel suits Dirichlet latent distributions.The geodesic distance is more sensitive near simplex boundaries than the common L-2 distance, which is important for sparse data.
- 4.2 Distribution matching: The training objective combines reconstruction loss with latent distribution matching, scaling reconstruction by 1/(s log V) to balance the two terms without another hyperparameter.The normalization is based on an uninformative decoder applied to a document of length s containing one unique word.
- 4.3 Adding noise: Mixing each encoder output with a random Dirichlet vector during training produces substantially better topics and lets the decoder train on stochastic topic representations.The mixture uses θ+ = (1 − α)θ + αθnoise, with α controlling the contribution of prior noise.
5 Topic extraction and TU measure
W-LDA extracts representative topic words from decoder weights and evaluates topic quality using both coherence and topic uniqueness. Topic uniqueness measures how rarely each topic’s top words recur across other topics, with higher values indicating greater diversity.
- Topic extraction: Representative words for topic k are the top entries of its decoder weight vector βk, sorted in descending order.The extracted top words are used to evaluate both coherence and uniqueness.
- TU measure: Topic uniqueness for topic k averages the reciprocal counts of its top words across all topics, so repeated words reduce the score.For each top-word position l, cnt(l,k) counts how many topics contain that word among their top words.
- TU measure: The average TU ranges from 1/K to 1, and higher values indicate more diverse topics.A word appearing only in its own topic contributes more than a word appearing across many topics.
6 Experiments and Results
Across synthetic and real-world evaluations, W-LDA recovers topics well and outperforms existing methods on coherence and uniqueness while retaining competitive document-classification accuracy. Ablations show that distribution matching and Dirichlet noise are important, and MMD is effective for matching the Dirichlet prior.
- Synthetic topic recovery: W-LDA recovers the ground-truth topics well in the synthetic LDA experiment, including the relative importance of most top words.The experiment uses 10,000 documents, five topics, and maximum precision over topic permutations for evaluation.
- Benchmark results: W-LDA achieves much higher NPMI and TU than existing methods, especially on NYTimes and Wikitext-103.Table 4 reports NPMI/TU for 50-topic benchmark comparisons.
- Document classification: W-LDA has competitive document-classification accuracy with LDA and NTM-R, whereas ProdLDA performs significantly poorly on DBpedia.Because W-LDA is not based on variational inference, the comparison uses test-set classification accuracy rather than ELBO-based perplexity.
- Distribution matching: Removing distribution matching generally causes latent-space collapse, with only one non-zero dimension and decoder weights that fail to produce meaningful topics.The ablation evaluates reconstruction loss without the distribution-matching term.
- Dirichlet parameter and noise effects: Adding Dirichlet noise with coefficient α=0.5 can significantly improve NPMI over α=0, while the effect of Dirichlet parameters 0.1 and 0.2 is mixed.The authors attribute the benefit to sparse noise spreading the latent space without washing out significant encoder signals.
- MMD versus GAN: GAN training encounters vanishing gradients for Dirichlet distribution matching, whereas the toy experiment compares GAN and MMD as alternative matching methods.The supplied passages report successful matching by both methods in the initial toy setup and motivate further comparison.
7 Conclusion and Future Work
W-LDA directly enforces a Dirichlet prior and improves topic coherence and diversity over existing topic models. GAN-based training remains unsuccessful, while richer priors are proposed as future work.
- W-LDA significantly improves topic quality in both coherence and diversity over existing topic models.
- MMD out-performs GAN for matching high dimensional Dirichlet distributions, and encoder noise can boost coherence without harming diversity.
- GAN-based W-LDA training was unsuccessful, motivating future work on newer GAN formulations for mode collapse and vanishing gradients.
- The W-LDA framework could support more complex priors, including a nested Chinese restaurant process for hierarchical topic models.
8 Appendix: synthetic topic recovery experiment details
The synthetic experiment uses an LDA-generated corpus with 10,000 documents, 100 vocabulary terms, five topics, and Dirichlet parameter 0.1.
- The synthetic corpus contains 10,000 LDA-generated documents with vocabulary size 100, five topics, and Dirichlet parameter 0.1.
- All models use five topics, while neural models use encoder networks with two hidden layers of 10 units each.
- Collapsed-Gibbs LDA runs for 2,000 iterations and Online LDA for 200 iterations using default parameters.
9 Appendix: additional TU and NPMI plots for W-LDA
Additional plots show that high NPMI can coexist with very low topic uniqueness when distribution matching is omitted, producing repetitive topics.
- Without distribution matching, Yelp P. achieves high NPMI but very low TU because its topics contain repetitive words.
- The repetitive Yelp P. topics include words such as “good”, “nice”, and “love”.
10 Appendix: document classification
The document-classification appendix evaluates latent document-topic vectors as features and reports that W-LDA is competitive with strong baselines. The accompanying topic-quality plots examine uniqueness and coherence across datasets and noise settings.
- Dirichlet parameter and noise effects: Figure 3 compares TU and NPMI across Dirichlet parameters and noise α for 20NG, NYTimes, and Wikitext-103.
- Dirichlet parameter and noise effects: Adding Dirichlet noise generally improves NPMI, whereas reconstruction-only training generally causes latent-space mode collapse and failed topic learning.
- Document classification: W-LDA latent vectors have competitive test classification accuracy with LDA variants and NTM-R.
- Document classification: ProdLDA performs significantly poorly on DBpedia, where its test and training document-topic distributions differ substantially.
11 Appendix: MMD vs GAN in distribution matching
The appendix compares GAN and MMD for matching Dirichlet latent distributions, finding that GAN works in low dimensions whereas MMD better captures high-dimensional modes.
- Both GAN and MMD match the 2D Dirichlet prior after training, as shown by their encoded-distribution histograms.The prior is a 2D Dirichlet distribution with parameter 0.1.
- MMD gradually captures more modes of the 50-dimensional Dirichlet prior, whereas GAN fails to match the prior.The comparison uses t-SNE plots of encoder outputs and prior samples across training epochs.
- The experiments compare encoder outputs with Dirichlet-prior samples across training epochs using histograms and t-SNE visualizations.GAN is shown at epochs 0, 10, 20, and 50 in the histogram experiment, while t-SNE uses epochs 0, 10, 30, and 99.
12 Appendix: topic words
This appendix lists topic-word outputs with topic identifiers, topic uniqueness, and NPMI values, illustrating topics spanning technical, scientific, cultural, religious, and sports-related themes.
- Each listed topic is accompanied by a topic ID, TU score, and NPMI score for evaluating topic diversity and coherence.The appendix explicitly states that the numbers at the beginning of each row encode topic ID, TU, and NPMI.
- The topic-word lists cover recognizable themes including computing hardware, religion, sports, astronomy, medicine, politics, and entertainment.Examples include hardware terms, religious vocabulary, hockey and racing terms, space-related words, medical terms, and film or television vocabulary.
- Some topic lists combine words from distinct domains, such as hockey with Armenian or Turkish historical terms and hardware with cycling vocabulary.These mixed lists appear among the reported topic-word outputs and their associated TU and NPMI values.