Source-linked AI summary
Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
Alexander Hoyle, Pranav Goel, Denis Peskov, Andrew Hian-Cheong, Jordan Boyd-Graber, Philip Resnik
TL;DR
Topic model evaluation relies heavily on automated coherence metrics, although their validation for neural topic models and the standardization of benchmarks remain limited. The paper combines a meta-analysis with standardized comparisons of classical and neural models using automated and human evaluations, finding that automated metrics can declare a winner when human judgments do not. It concludes that fully automatic evaluation is not reliably sufficient for model comparison.
Problem
Automated coherence was validated mainly with classical models, while neural models and inconsistent benchmarking practices leave its relationship to human judgments uncertain.
Method
The paper conducts a literature meta-analysis and standardized evaluations of one classical and two neural topic models using human ratings, word intrusion, and automated metrics.
Results
Automated evaluation declares winners between models when corresponding human evaluations cannot.
Takeaways & Limitations
Fully automated topic model evaluation should not be treated as independently decisive when selecting among models.
Takeaways & Limitations
Human judgment is itself uncertain, making it difficult to determine when an increase in automated coherence implies a significant improvement in human scores.
Abstract
from arXiv · showhide
Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrences in a reference corpus. Contemporary neural topic models surpass classical ones according to these metrics. At the same time, topic model evaluation suffers from a validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, a meta-analysis of topic modeling literature reveals a substantial standardization gap in automated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets. Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments.
1 Revisiting Topic Model Evaluation
Topic model evaluation has shifted from human-centered judgments to automated coherence metrics, but neural models expose unresolved validation and standardization gaps. The paper addresses these gaps and finds that automated evaluations can select winners when human evaluations cannot.
- Topic models are widely used by domain experts to identify global categories or themes in document collections.
- Automated coherence metrics became prominent after held-out perplexity disagreed with human judgments of interpretability.
- Neural topic models differ qualitatively from classical models, so correlations between automated metrics and human judgments established for classical models may not apply.
- Evaluation practice varies across datasets, preprocessing, hyperparameter optimization, and metric specification, limiting the generalizability and reliability of findings.
- The paper standardizes datasets and model optimization, evaluates classical and neural models with human ratings and word intrusion, and examines automated-human correlations.
- Automated evaluation declares winners between models when corresponding human evaluations cannot.
2 Operationalizing Topic Coherence
Topic coherence is treated as human recognition of an identifiable category, while automated coherence estimates word relatedness from corpus co-occurrences. The paper questions whether these automated measures remain valid for neural topic models.
- A topic model uses latent topics to summarize a document collection, with topics represented as distributions over word types.
- Neural topic models use continuous word representations and gradient optimization, and claim greater interpretability than earlier models.
- Real-world coherence is an intangible human-recognized sense that a set of terms enables identification of a category.
- Human assessment uses topic rating and word intrusion, where raters score topic quality or identify a word that does not belong.
- NPMI compares joint and marginal probabilities of top-word pairs, estimated from co-occurrence counts in a reference corpus and context window.
- Automated coherence has historically been validated against human judgments, but the applicability of classical-model correlations to neural models remains uncertain.
3 A Meta-Analysis of Neural Topic Modeling
The meta-analysis finds substantial inconsistency in neural topic model evaluation and incomplete reporting of automated metric calculations. These practices make metrics and model comparisons difficult to reproduce and may limit their generalizability.
- The survey examines forty neural topic modeling papers that claim superior topic coherence while comparing models against existing baselines.
- 30% of papers use inconsistent preprocessing across datasets within the same paper, and many omit details needed to reproduce the pipeline.
- 40% of papers fail to clearly specify model tuning procedures, often omitting even the metric used for model selection.
- Three out of four papers fail to explicitly identify the reference corpus used to calculate automated coherence.
- Researchers sometimes use bespoke implementations or context-window settings that differ from human-validation studies, including document-wide NPMI windows.
- Claims of coherence improvement depend on whether validity results from Lau et al. (2014) generalize to neural topic models.
4 Closing the Standardization Gap for Topic Models
The paper standardizes topic-model comparisons by applying a common preprocessing pipeline and fair tuning procedure to two datasets and three models. It evaluates one classical baseline alongside two neural models while preserving reproducibility.
- 4.1 Datasets and Preprocessing: The study processes two commonly used datasets with a standardized, portable pipeline and supports reproducible end-to-end comparisons.The pipeline mandates consistent preprocessing between training and reference corpora and supports multi-word expressions during vocabulary creation.
- 4.1 Datasets and Preprocessing: Reference counts come from identically processed Wikipedia and New York Times corpora used to calculate automated coherence.The reference resources contain 4.6M Wikipedia documents and 1.8M New York Times documents.
- 4.1 Datasets and Preprocessing: Preprocessing tokenizes text, identifies entities as multi-word tokens, avoids lemmatization, and removes common and corpus-specific stopwords.Corpus-specific stopwords are tokens appearing in more than 90% of documents.
- 4.2 Models: The comparison includes Gibbs-LDA as the classical baseline, Dirichlet-VAE as a state-of-the-art neural model, and ETM as an embedding-based neural model.Dirichlet-VAE is reimplemented, while ETM incorporates word embeddings in its generative model.
- 4.2 Models: Models receive identical preprocessing, model-selection criteria, and hyperparameter-tuning treatment to support computationally fair head-to-head evaluation.ETM uses a fixed computational budget and 164 randomly selected hyperparameter settings across datasets for each model type.
5 Human Evaluations of Topic Quality
The study evaluates topic quality with crowdworker ratings and word-intrusion judgments under statistically powered, controlled task designs. Human judgments are more nuanced than automated comparisons, showing greater variability over a smaller score range.
- Human Evaluation Design: Crowworkers complete ratings and word-intrusion tasks through Prolific and Qualtrics, with compensation calibrated to 15 USD/hour.Workers receive 2.5 USD for ratings surveys and 3 USD for word-intrusion surveys.
- Human Evaluation Design: At least fifteen crowdworkers per topic are recruited to achieve statistical power of at least 1−β = 0.9 at α = 0.05.The sample-size target is selected using a generative model of annotations.
- Task Construction: Each dataset contributes fifty topics per model; intrusion items contain five top words plus one intruder, while ratings show the top ten words in order.The ratings task also includes sixteen synthetic poor-quality topics for calibration and respondent filtering.
- Evaluation Results: Human judgments vary more across a narrower score range than automated NPMI evaluations, which suggest a clearer model winner.Figure 2 uses pairwise one-tailed significance tests at α = 0.05 to compare model scores.
- Task Construction: Workers report familiarity with topic words because esoteric topics may otherwise receive spuriously low human scores.The familiarity question is intended to distinguish unfamiliarity from poor topic coherence.
6 Human Judgment Differs From Automated Metrics
Human evaluations distinguish the models differently from automated metrics, while automated metrics can exaggerate meaningful model differences. The discrepancy is partly associated with esoteric, corpus-specific topics and persists after filtering unfamiliar terms.
- Human evaluation: D-VAE performs better on word intrusion, whereas ratings favor G-LDA.Pairwise tests use a proportion test for intrusion scores and a Mann–Whitney U test for ratings.
- Human evaluation: 2.5 to 2.8 ratings and 0.7 to 0.8 intrusion means align with prior human-evaluation results.Median task completion time was 8–9 minutes, and annotator agreement was 0.75.
- Automated evaluation: NPMI ranks D-VAE first and G-LDA second, contradicting the human judgments.Wikipedia appears best correlated with human judgments, including for models trained on NYT data.
- Automated evaluation: Automated metrics often overstate model differences, with even the best metrics producing false significant differences roughly one-fifth of the time.The analysis bootstraps two independent sets of K = 50 topics and evaluates false discovery rates against human-score differences.
- Explaining the discrepancy: NPMI and Cv scores negatively correlate with familiarity, while D-VAE has familiar-word rates near 70% versus over 90% for G-LDA and ETM.Filtering unfamiliar respondents substantially improves D-VAE’s intrusion accuracy and “Very related” ratings, indicating narrower, more esoteric topics.
- Explaining the discrepancy: Even after filtering unfamiliar respondents, automated metrics still overstate model differences, so human evaluation choices may also remain problematic.The paper notes that topics with high NPMI but low human ratings can be corpus-specific or fail to co-occur within NPMI’s ten-word window.
7 So...is Automated Topic Modeling Evaluation Broken?
The authors argue that topic-model evaluation needs reconsideration because automated and human evaluations are abstractions of real-world user needs. They propose future metrics that better approximate those preferences while noting neural models’ computational demands.
- Topic-model evaluation, both automated and human, is overdue for careful reconsideration.
- Generic evaluation with domain-general corpora is unsuitable because content analysis has neither a generic corpus nor a generic analyst.
- Neural models usually run longer than classical alternatives, creating a practical constraint for practitioners using local, CPU-only environments.
A.1 List of Neural Topic Modeling Works used in our Meta-Analysis
The meta-analysis comprises forty publications drawn from a survey of neural topic models.
- 40 publications were included in the meta-analysis, sourced from a survey of neural topic models.
A.2 Preprocessing Details
The preprocessing pipeline standardizes tokenization, filtering, entity handling, document lengths, and vocabulary construction across datasets and coherence references.
- Document processing: Documents with fewer than 25 whitespace-separated tokens are excluded before processing, and those with fewer than five tokens afterward are removed.
- Document processing: Documents are truncated to 5,000 tokens for NYT and 19,000 for WIKI, affecting less than 0.15% of documents in either dataset.
- Vocabulary creation: Terms are tokenized and lowercased with spaCy, without lemmatization; selected noun entities are joined with underscores.
- Vocabulary filtering: The vocabulary comes from training data, while coherence reference texts receive identical processing and use the same vocabulary.
- Vocabulary filtering: Vocabulary filtering removes stopwords, tokens of two or fewer characters, tokens failing a regular expression, and terms appearing in over 90% of documents.
A.3 Training Details
The study documents reproducible training and evaluation infrastructure, including model-specific tuning, corpus statistics, implementation choices, and compute environments for one classical and two neural models.
- Reproducibility: The implementation releases scripts covering the models’ hyperparameter configurations and documents corpus statistics and the spaCy version used.
- Hyperparameter tuning: All three models are tuned around reported optimal settings, including two training-iteration or epoch values per model.
- Meta-analysis: The evaluation meta-analysis records whether papers specify human evaluation, perplexity, coherence, implementation, reference corpus, preprocessing, search, error bars, and LDA baselines.
- Datasets: The datasets differ in domain, average document length, and document count, with WIKI from Merity et al. and NYT from Sandhaus.
- Model training: G-LDA tunes α and β as smoothing parameters, while D-VAE and ETM vary their respective priors, learning rates, regularization, annealing, and training settings.
- Compute: Neural models run on NVIDIA T4 GPUs, whereas G-LDA runs CPU-only on AWS instances with specified memory and CPU allocations.
A.4 Instructions for Crowdworkers
The appendix documents standardized crowdworker recruitment, topic-rating and word-intrusion tasks, power analyses, and evaluation caveats. It also reports that automated metrics can misdiagnose results and that correlation estimates vary with rating-scale design.
- A.4 Instructions for Crowdworkers: Participants were recruited through Prolific.co after a pilot found higher inter-worker agreement than Mechanical Turk, with U.S. residence and English fluency requirements.Respondents were paid at a rate corresponding to $15 an hour.
- A.4 Instructions for Crowdworkers: Hyperparameter tables specify settings for G-LDA, D-VAE, and ETM, with best settings selected using NPMI from a 10-token reference-corpus window.Runtime comparisons distinguish CPU-only G-LDA from neural models using a single GPU.
- A.5 Power Analysis for Human Evaluation Tasks: The ratings task asks workers to judge how related topic words are and how familiar they are with those terms.The examples use a three-point scale: Not Very Related, Somewhat Related, and Very Related.
- A.5 Power Analysis for Human Evaluation Tasks: The word-intrusion task asks workers to identify the word whose meaning or usage is most unlike the other words in each set.Workers are shown ten sets of six words and are instructed to choose the most out-of-place word when multiple candidates seem unrelated.
- A.5 Power Analysis for Human Evaluation Tasks: 15 annotators per topic for ratings and 25 for intrusion were selected by power analysis with α = 0.05 and desired power 1 −β = 0.9.The simulations modeled noisy three-point ratings and binary intrusion responses.
- A.5.1 Power analysis for equivalence: ϵ = 0.05 for intrusion and ϵ = 0.11 for ratings define non-inferiority bounds corresponding to roughly 2.5 incoherent topics for both tasks.The analysis tests when differences between human and automated scores are not meaningful.
- A.7 Filtering on Term Familiarity: Automated metrics still misdiagnose significant results at a rate of one in six even in the best case, while filtering unfamiliar respondents motivates accounting for domain expertise and information need.Unfamiliarity can lower mean ratings for specialized topics, such as fungi-related terminology.
- A.8 Five-point Ratings Scale: Using a five-point ratings scale broadly reduces correlation values relative to the commonly used three-point scale, and the study cautions that English-only findings may not transfer directly to other languages.The paper also questions whether topic models encourage broad, generic readings rather than engagement with textual specifics.