Source-linked AI summary
Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge
Ryan J. Gallagher, Kyle Reing, David Kale, Greg Ver Steeg
TL;DR
The paper addresses the complexity of generative topic models and their extensions for human input. It introduces CorEx, an information-theoretic model that learns maximally informative topics without generative assumptions and incorporates domain knowledge through anchor words. Across datasets and metrics, CorEx and anchored CorEx produce topics comparable in quality to LDA-based methods, while anchored CorEx can guide themes that do not naturally emerge.
Problem
Generative topic models require increasingly elaborate assumptions to incorporate metadata or human input, constraining topics to narrower predefined views.
Method
CorEx learns topics by maximizing information about documents, using an information-theoretic objective and flexible anchor-word constraints for domain knowledge.
Results
CorEx and anchored CorEx produce topics comparable in quality to unsupervised and semi-supervised LDA variants across datasets and metrics.
Takeaways & Limitations
Anchored CorEx can guide topic models toward themes that do not naturally emerge and supports different strategies for separability, representation, and topic aspects.
Takeaways & Limitations
CorEx uses binary rather than standard count data and requires each word to appear in only one topic, with longer documents more affected by the count-data limitation.
Abstract
from arXiv · showhide
While generative models such as Latent Dirichlet Allocation (LDA) have proven fruitful in topic modeling, they often require detailed assumptions and careful specification of hyperparameters. Such model complexity issues only compound when trying to generalize generative models to incorporate human input. We introduce Correlation Explanation (CorEx), an alternative approach to topic modeling that does not assume an underlying generative model, and instead learns maximally informative topics through an information-theoretic framework. This framework naturally generalizes to hierarchical and semi-supervised extensions with no additional modeling assumptions. In particular, word-level domain knowledge can be flexibly incorporated within CorEx through anchor words, allowing topic separability and representation to be promoted with minimal human intervention. Across a variety of datasets, metrics, and experiments, we demonstrate that CorEx produces topics that are comparable in quality to those produced by unsupervised and semi-supervised variants of LDA.
1 Introduction
Topic modeling has largely relied on generative assumptions that become increasingly elaborate when incorporating metadata or human input. CorEx instead learns informative topics without specifying a data-generating model, while supporting extensions and domain-guided anchoring.
- Generative topic models specify mechanisms for how documents are written to infer latent topics, including LDA and matrix factorization techniques.
- Generalizing these models to incorporate metadata introduces elaborate assumptions that can constrain topics to narrower predefined views.
- CorEx learns maximally informative topics through an information-theoretic framework without assuming a particular data-generating model.
- CorEx naturally extends to hierarchical and semi-supervised variants without additional modeling assumptions.
- Domain words can guide CorEx toward underrepresented topics, including rare diseases, aid-relevant news themes, and aspects of complex social issues.
- The paper introduces efficient sparse-data optimization, anchor-word integration, and evaluations showing CorEx variants comparable in quality to unsupervised and semi-supervised LDA variants.
2 Methods
CorEx learns latent topics by maximizing how much they explain word dependencies, without assuming a particular data-generating model. Its implementation uses iterative optimization, sparse binary-document computations, and an information-bottleneck formulation for flexible anchor-word supervision.
- CorEx objective: Total correlation measures multivariate dependence and is zero exactly when the joint distribution factorizes.Conditioning on a topic reduces total correlation; if the conditional total correlation is zero, the topic explains all dependencies in the word group.
- CorEx objective: CorEx seeks latent topics that maximally explain dependencies among word types by maximizing a lower bound on total correlation.In topic modeling, word variables form groups X_G, while binary latent topics Y_1,...,Y_m explain their dependencies.
- Optimization: CorEx optimizes randomly initialized parameters through iterative updates while constraining word groups to be non-overlapping.The implementation relaxes hard membership with a softmax controlled by λ, gradually increasing λ to enforce sharper competition among latent factors.
- Interpretation: After convergence, mutual-information terms rank informative words for each factor, while pointwise total correlation identifies samples particularly informative about specific factors.The expected log normalization constant provides a free estimate of each factor’s objective contribution.
- Sparse optimization: Sparse optimization changes CorEx’s asymptotic scaling from O(Nn) to O(n)+O(N)+O(ρ), where ρ is the number of nonzero data entries.The speedup exploits binary documents by treating absent words as the default contribution and correcting only present-word terms.
- Anchor words: Anchoring constrains α_i,j to β_i,j, with β_i,j ≥ 1 controlling anchor strength, and supports multiple words or topics in flexible combinations.This connects CorEx to the information bottleneck by making selected words relevance variables for chosen topics.
3 Related Work
CorEx’s anchoring approach builds on prior methods that use word-level information or logical constraints to guide topic discovery. It differs by allowing words to anchor multiple topics and by connecting to latent-tree and surprising-document analyses.
- Anchored topic models: Anchor-word methods use high-precision markers or manually inserted metadata to guide topics in semi-supervised models.CorEx treats anchor words as fuzzy logic markers embedded in the topic model.
- Anchored topic models: Unlike z-label models, CorEx allows one word to anchor multiple topics, supporting topics organized around different word senses.z-labels restrict known topic-specific words to subsets of possible topics and require effort to identify differing senses.
- Logical constraints: Must-Link and Cannot-Link constraints enforce topic separability, but they address partitioning muddled topics less directly than anchoring words to help topics emerge.These constraints specify relationships between words rather than directly anchoring a word or word set to a topic.
- Related formulations: CorEx mathematically resembles latent-tree topic models, and both approaches have reported better perplexity than hierarchical Dirichlet-process and Chinese-restaurant-process models.CorEx has also been studied for finding surprising documents.
4 Data and Evaluation Methods
The evaluation uses humanitarian, clinical, and general-news corpora with domain lexicons where available, then compares topic models using coherence, classification, and clustering measures. Anchored CorEx is evaluated through automatically derived label-associated anchor words and Figure 2’s cross-dataset baseline comparison.
- Datasets: The HA/DR corpus contains 504,000 articles covering 21 disaster types and is paired with an approximately 34,000-word domain lexicon.The lexicon begins with expert and CrisisLex seed terms, then expands through word embeddings and filtering.
- Datasets: The clinical corpus contains 1,237 deidentified discharge summaries labeled by experts with 15 obesity-associated conditions.A text pipeline extracts 3,231 clinical term types, and the documents are represented as bags of words and phrases.
- Datasets: The evaluation also uses the preprocessed 20 Newsgroups dataset from Scikit-Learn.
- Evaluation metrics: Topic quality is measured with UMass coherence, while document classification uses multiclass logistic regression with a 60/40 train-test split.Topic distributions provide the document-classification features.
- Evaluation metrics: Clustering assigns each document to its highest-probability topic and evaluates within-cluster homogeneity and across-cluster adjusted mutual information.Clinical notes omit clustering metrics because their multi-label structure makes these measures not well-defined.
- Anchoring and comparison: For each dataset label, anchor words are selected as words with the highest mutual information with that label.Word presence is considered for documents carrying the label; Figure 2 compares CorEx and LDA as topic counts vary across three datasets.
5 Results
CorEx and anchored CorEx generally match or exceed LDA across classification, clustering, and coherence, while anchoring improves label-relevant separation and representation. Anchored CorEx also compares favorably with semi-supervised LDA variants and supports interpretable exploration of complex issue frames.
- LDA baseline comparison: Across three datasets, CorEx topics yield document classification results on par with or better than LDA topics.
- LDA baseline comparison: CorEx consistently produces higher-homogeneity document clusters than LDA, nearly doubling LDA homogeneity on disaster relief articles.
- LDA baseline comparison: CorEx outperforms LDA in topic coherence on two of three datasets, although LDA is more coherent on clinical health notes.
- Anchoring for topic separability: Across all measures, CorEx and anchored CorEx outperform LDA, while anchoring improves homogeneity and adjusted mutual information without changing topic coherence.
- Anchoring for topic separability: Anchored CorEx has clustering similar to or better than z-labels LDA and must/cannot link LDA, but is less coherent than both on disaster relief articles.
- Anchoring for topic representation: Anchoring increases topic overlap as β rises, introducing about two relevant words for many disaster types and up to four for clinical health notes.
- Anchoring for topic representation: Anchored topics are often more coherent and can reach 1.5 to 2 times pre-anchoring coherence, especially for underrepresented labels.
- Anchoring for topic representation: Anchoring produces mixed disaster-label F1 changes but raises clinical-health-note F1 by about 0.1 on average and up to 0.3 for some diseases.
6 Discussion
CorEx replaces LDA’s generative assumptions with an information-theoretic objective, while anchored CorEx flexibly guides topics toward user-selected themes. The framework supports multiple anchoring strategies and produces topics comparable in quality to LDA-based methods, with technical efficiency-related limitations.
- Core model: CorEx seeks maximally informative topics without relying on LDA-based generative assumptions.Its objective is based on total correlation, and it uses binarized word counts.
- Anchored CorEx: Anchored CorEx incorporates word-level domain knowledge through the information bottleneck to guide themes that may not naturally emerge.The approach can produce more coherent and predictive topics.
- Performance: CorEx and anchored CorEx consistently produce topics comparable in quality to LDA-based methods.The comparison is reported despite CorEx using only binarized word counts.
- Anchoring strategies: Topic separability, representation, and aspect discovery can be promoted through distinct anchoring strategies.Disjoint word groups can be assigned to separate topics, one group can be emphasized for a single topic, or one group can be assigned to multiple topics.
- Limitations: CorEx’s sparse optimization relies on binary rather than standard count data and restricts each word to one topic, especially affecting longer documents.Chunking longer documents into shorter subdocuments can partly address the count-data limitation; the restrictions are computational rather than theoretical.
- Implications: The lightweight framework leaves possibilities for theoretical extensions and applications that help domain experts guide topic models with minimal intervention.The supported use case is capturing otherwise eclipsed themes in documents.
A Supplemental Material: Anchor Words and Topic Examples
The supplemental material documents hierarchical CorEx visualizations and ranked topic outputs for disaster-relief articles and clinical health notes. It also lists words with the highest mutual information for disease and disaster types.
- Anchor words: Tables A1 and A2 list words with the highest mutual information for disease and disaster types.
- Hierarchical models: Figures A1 and A2 show hierarchical CorEx topic models for disaster-relief articles and clinical health notes.Edge widths are proportional to mutual information with the latent representation.
- Disaster-relief topics: Tables A3 and A4 present topics 1–50 from the best of 10 CorEx models on disaster-relief articles, ranked by total correlation explained.
- Clinical-health topics: Tables A5 and A6 present topics 1–30 from the best of 10 CorEx models on clinical health notes, ranked by total correlation explained.