Source-linked AI summary
Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression
David Mimno, Andrew McCallum
TL;DR
Existing generative topic models are awkward for combining text with document metadata. The paper introduces DMR, which makes document-topic priors functions of observed features, and reports comparable or improved performance against specialized models. DMR also keeps sampling no more complicated than a simple LDA sampler while supporting flexible metadata conditioning.
Problem
Existing metadata-rich topic models are either specialized for particular data types or more complicated, making combinations of text and metadata awkward to model.
Method
DMR places a log-linear prior on document-topic distributions whose Dirichlet parameters are functions of observed features such as authors, venues, references, and dates.
Results
DMR achieves similar or improved performance to previously published topic models designed for specific metadata types, with performance considerably better in some comparisons.
Takeaways & Limitations
DMR provides an off-the-shelf way to develop topic models with arbitrary features while retaining relatively simple inference and little additional programming work.
Takeaways & Limitations
DMR complements, rather than replaces, generative models such as AT and sLDA, which support inferences about hidden variables and integration into more complicated hierarchical models.
Abstract
from arXiv · showhide
Although fully generative models have been successfully used to model the contents of text documents, they are often awkward to apply to combinations of text data and document metadata. In this paper we propose a Dirichlet-multinomial regression (DMR) topic model that includes a log-linear prior on document-topic distributions that is a function of observed features of the document, such as author, publication venue, references, and dates. We show that by selecting appropriate features, DMR topic models can meet or exceed the performance of several previously published topic models designed for specific data.
1 Introduction
DMR addresses the difficulty of combining text with metadata by conditioning document-topic priors on arbitrary observed features. It can emulate metadata-specific topic models while retaining relatively simple inference and comparable or better performance.
- Motivation: Metadata-rich topic modeling can reveal author profiles and topic trends, but existing models often require specialized designs for particular metadata types.Generative models incorporate metadata by generating it from topics or by using metadata-specific mixtures of topic distributions.
- DMR approach: DMR conditions topic priors on observed document features, allowing users to specify arbitrarily complicated combinations of metadata.Examples of usable features include authors, venues, references, and dates.
- Related approaches: Unlike downstream models such as sLDA, DMR uses observed features through a log-linear model rather than generating metadata from topic variables.sLDA specifies a generalized linear model for each metadata type, whereas DMR conditions on the features.
- DMR approach: DMR models can incorporate continuous, discrete, and categorical features without additional coding while retaining relatively simple inference.This flexibility contrasts with specialized models and more complicated approaches whose inference can become intractable.
- Results: Across comparisons with models designed for specific metadata, DMR performance is in almost all cases comparable and can be considerably better.The comparisons use DMR features chosen to emulate the competing models.
2 Modeling the influence of document metadata with Dirichlet-multinomial regression
The DMR model represents document metadata through feature-dependent Dirichlet parameters for document-topic distributions. Its training alternates Gibbs sampling of topic assignments with numerical optimization of regression parameters.
- Feature representation: Each document is represented by a feature vector encoding observed metadata, with an intercept default feature included to model each topic’s mean level.Author indicators, for example, place 1s for listed authors and 0s elsewhere.
- Generative process: Each topic has a feature-length parameter vector, and the generative process uses these parameters with the document feature matrix.The topic-specific vectors determine how document features affect the topic prior.
- Model parameters: The model includes σ2, β, and |T| as fixed parameters controlling parameter-prior variance, topic-word smoothing, and the number of topics.β is the Dirichlet prior on topic-word distributions.
- Inference: Integrating over multinomial topic-word distributions yields a complete log likelihood for topic assignments, whose derivatives support optimizing the regression parameters.The derivative is taken with respect to λtk for topic t and feature k.
- Model structure: In the graphical model, the document-specific topic prior α is a function of observed metadata feature values.Different combinations of feature values produce different topic priors.
- Inference: Training alternates sampling topic assignments conditioned on words and features with numerical optimization of λ using stochastic EM.The implementation combines L-BFGS optimization with Gibbs sampling-based LDA training.
3 Related Work
DMR differs from prior conditional and generative approaches by accepting arbitrary observed features without additional model specification. It uses a conditional formulation while remaining applicable to complex multimodal metadata.
- Conditional versus generative models: sLDA uses topic variables to generate observed metadata through a log-linear model, whereas DMR conditions on observed features to model topic variables.The two models place the log-linear component in opposite directions.
- DMR flexibility: Off-the-shelf DMR models can be applied to any set of features without additional model specification.Complex, multimodal, dependent features are no more difficult to train than a single observed real-valued feature.
- Statistical background: Dirichlet-multinomial regression also appears in economics as an overdispersed generalized linear model, without a mixture model or hidden variables.In that setting, it is equivalent to logistic regression with extra-multinomial variance.
- Statistical background: DMR produces unnormalized Dirichlet parameters that can serve as priors for Bayesian mixture models.This property distinguishes its role from normalized multinomial parameterizations.
4 Experimental Results
The experiments compare DMR with metadata-specific topic models and LDA across author, citation, and date features. DMR is generally comparable to or better than specialized generative models while retaining flexible feature conditioning.
- Experimental setup: DMR models emulate previously published metadata-specific topic models by conditioning on features such as authors, citations, and dates.The experiments use features designed to reproduce the behavior of specialized models, rather than arguing that those models should be replaced.
- Experimental setup: The evaluation uses research papers with text, publication years, venues, author IDs, and references, filtered to artificial-intelligence venues and metadata with sufficient frequency.Metadata-specific models are trained only on documents containing the relevant metadata.
- Evaluation: DMR is evaluated with held-out perplexity and empirical likelihood, which assess topic-word quality and prediction of document-level topic mixtures.Empirical likelihood uses unconditional sampled topic distributions, whereas perplexity also uses local information from the test document.
- Author features: For author features, DMR achieves much better perplexity than LDA or Author-Topic, while Author-Topic is consistently slightly better than DMR in empirical likelihood.Both author-aware models substantially outperform LDA in empirical likelihood; the reported perplexity advantage is attributed to DMR’s fresh document-specific Dirichlet prior.
- Metadata prediction: DMR ranks held-out authors consistently higher than Author-Topic and has citation-prediction performance comparable to the generative citation topic model.The author-ranking comparison uses rankings rather than directly comparable likelihood values between DMR and Author-Topic.
- Citation features: With citation features, the Citation model has slightly better empirical likelihood and significantly better perplexity than DMR, while both outperform LDA in empirical likelihood.The authors suggest citation topic multinomials can adapt more easily because each may contain only a few dozen words.
5 Conclusions
DMR provides a flexible way to build topic models from arbitrary document features while retaining relatively simple inference. It can emulate prior models with similar or improved performance and may offer sampling-efficiency advantages.
- DMR can emulate many previously published topic models with similar or improved performance and little additional modeling or programming work.Across comparisons, performance was almost always comparable to similar generative models and sometimes considerably better.
- DMR incorporates arbitrary document features while keeping inference as simple as a standard LDA sampler.Its document metadata is absorbed into document-specific Dirichlet parameters, avoiding additional hidden variables in the sampling phase.
- DMR substantially improves perplexity over TOT and also improves empirical likelihood when trained with date features.TOT has perplexity roughly equivalent to LDA, whereas DMR performs substantially better on perplexity and exceeds TOT in empirical likelihood.
- DMR’s faster sampling phase can offset parameter-optimization overhead, especially when many sampling iterations occur between optimizations.The optimization overhead decreases as the model converges.
- DMR complements generative models such as AT and sLDA rather than replacing their hidden-variable inference and hierarchical-model capabilities.The paper identifies hybrid sLDA-DMR models as future work.