Source-linked AI summary
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
Federico Bianchi, Silvia Terragni, Dirk Hovy
TL;DR
Topic models often produce incoherent word groups, and Bag-of-Words representations omit syntactic and semantic relationships relevant to coherence. The paper extends Neural ProdLDA with contextualized document embeddings, finding significantly more coherent topics and competitive diversity, while noting a limitation from pre-processed datasets.
Problem
Topic models often produce incoherent word groups, while Bag-of-Words representations disregard syntactic and semantic relationships among document words.
Method
The paper extends Neural ProdLDA by incorporating contextualized document embeddings into neural topic modeling.
Results
The approach significantly improves topic coherence and produces competitive topic-diversity results.
Takeaways & Limitations
The results suggest that topic models benefit from latent contextual information missing in Bag-of-Words representations.
Takeaways & Limitations
Using already pre-processed datasets can be sub-optimal.
Abstract
from arXiv · showhide
Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.
1 Introduction
Topic models are difficult to interpret when their discovered word groups lack coherence, while Bag-of-Words inputs omit syntactic and semantic relationships. The paper addresses this gap by incorporating contextualized representations into neural topic models.
- Coherent topics are easier to interpret and are considered more meaningful.
- Bag-of-Words representations disregard syntactic and semantic relationships among document words, representing inputs in an inherently incoherent manner.
- Pre-trained language models provide contextualized word and sentence representations that have advanced state-of-the-art performance across many NLP tasks.
- The paper extends Neural ProdLDA to include contextualized representations.
- The proposed method produces significantly more coherent topics and suggests that topic models benefit from latent contextual information missing in Bag-of-Words representations.
2 Neural Topic Models with Language Model Pre-training
CombinedTM incorporates contextualized document embeddings into a neural topic-modeling architecture built around ProdLDA and SBERT. It combines contextual representations with Bag-of-Words inputs, while documents exceeding SBERT’s sentence-length limit lose their remaining text.
- CombinedTM combines the neural topic model ProdLDA with SBERT embedded representations.
- The method is agnostic to the topic model and pre-trained representations when the topic model extends an autoencoder and the representations embed documents.
- ProdLDA uses a Variational AutoEncoder whose inference network maps Bag-of-Words documents to continuous latent representations and whose decoder reconstructs the Bag-of-Words.
- SBERT document embeddings are projected through a hidden layer and concatenated with the Bag-of-Words representation.
- Documents longer than SBERT’s sentence-length limit lose the remainder of their text.
3 Experimental Setting
The experiments evaluate topic models across five datasets using two coherence metrics and one topic-diversity metric. Models are compared under similar hyper-parameter configurations across five topic counts and 30 runs, with contextualized representations generally yielding the strongest coherence.
- 3.1 Datasets: The evaluation uses five datasets: 20NewsGroups, Wiki20K, Tweets2011, Google News, and StackOverflow.
- 3.1 Datasets: Using already pre-processed datasets can be sub-optimal, although many datasets in the literature are already pre-processed.
- 3.2 Metrics: The models are evaluated with normalized pointwise mutual information, an external word-embedding coherence measure, and inversed rank-biased overlap for topic diversity.
- The comparison includes ProdLDA, NVDM, ETM, MetaLDA, and LDA, trained with similar hyper-parameter configurations.
4 Results
CombinedTM produces the most coherent topics across corpora and topic settings while maintaining competitive diversity, and contextualized encoder choice affects coherence.
- Quantitative Evaluation: Across all corpora and topic settings, CombinedTM provides the most coherent topics while maintaining competitive topic diversity.Metrics were averaged over 30 runs for 25, 50, 75, 100, and 150 topics.
- Quantitative Evaluation: CombinedTM outperforms comparison models on coherence, while MetaLDA is generally the second strongest model.LDA and NVDM obtain low coherence; ETM has weaknesses on short texts and topic similarity, while MetaLDA benefits from pretrained word embeddings.
- Quantitative Evaluation: On 4 of 5 datasets, CombinedTM achieves the best τ coherence, while remaining highly competitive on 20NG, where MetaLDA performs best.Table 3 compares average τ across different topic counts, with results averaged over 30 runs and significance marked by t-tests.
- Quantitative Evaluation: Examples of top words in the supplementary materials illustrate the increased coherence obtained with SBERT embeddings.These examples provide qualitative support for the quantitative evaluation.
- Using Different Contextualized Representations: RoBERTa fine-tuned on STSb substantially increases coherence relative to the alternative contextualized encoder evaluated in CombinedTM.The comparison uses stsb-roberta-large and bert-base-nli-means representations, with results averaged across datasets.
5 Related Work
Neural topic models have attracted growing interest because of their flexibility and scalability, with prior work using neural networks and variational inference.
- Neural Topic Models: Neural topic models have gained increasing success and interest because of their flexibility and scalability.Prior approaches use neural networks or neural variational inference, including NVDM and related models.
6 Conclusions
The paper proposes a simple method for incorporating contextualized embeddings into topic models and reports significantly improved topic quality, highlighting context as important for topic modeling.
- Conclusions: The paper proposes a straightforward method for incorporating contextualized embeddings into topic models.The authors characterize the method as simple and straightforward.
- Conclusions: The proposed model significantly improves the quality of discovered topics.The conclusion identifies contextual information as a significant element for topic modeling.
Ethical Statement
The study uses datasets from recent literature and does not use or infer sensible information; the authors assess the risk of model abuse as low.
- Ethical Statement: The study uses datasets from recent literature and does not use or infer sensible information.The ethical statement describes the data and information-handling scope of the research.
- Ethical Statement: The authors assess the risk of possible abuse of their models as low.This is the paper’s stated risk assessment.
A Datasets
The study preprocesses 20NewsGroup and Wiki20K by removing several token types and limiting the BoW vocabulary to 2,000 frequent words.
- 20NewsGroup and Wiki20K were preprocessed for the experiments.
- The preprocessing removed punctuation, digits, and NLTK English stop-words.
- The BoW representation retained only the 2,000 most frequent words in each document collection.
B.1 ProdLDA
The experiments use established implementations and broadly matched training settings, while the combined model augments ProdLDA with SBERT features concatenated to the BoW.
- The combined model and ProdLDA use one hidden layer with 100-dimensional softplus units in the inference network.
- The combined model is trained for 100 epochs with ADAM, while other implementations use their authors’ or default configurations where specified.
- The combined model adds SBERT English embeddings to the BoW through a learnable dense mapping to the vocabulary size.
- Experiments were run on two laptops with GeForce GTX 1050 GPUs, 16GB RAM, and CUDA 10.0.
C.1 Runtime
Runtime is evaluated per epoch for CTM and ProdLDA across two datasets and two topic counts; vocabulary size is identified as the main computational factor.
- Vocabulary size most strongly influences computational time.
- Table 5 compares one-epoch runtimes for CTM and ProdLDA on Google News and 20Newsgroups with 25 and 50 topics.
- ProdLDA runs faster than CTM because CTM includes an added representation.
- The reported runtimes are considered similar enough to support use of CTM with common hardware.