Source-linked AI summary
Supervised Topic Models
David M. Blei, Jon D. McAuliffe
TL;DR
The paper addresses prediction for labelled documents by introducing sLDA, which jointly models text and external responses through supervised latent topics. It develops variational approximate maximum-likelihood estimation and tests the model on movie ratings and Senate amendment tone, finding improved predictive performance over unsupervised LDA regression and lasso.
Problem
Predictive topic modeling requires topics aligned with an external document response rather than only with dominant word patterns in the corpus.
Method
sLDA jointly models document words and responses, using latent empirical topic frequencies in a generalized linear model and variational methods for approximate inference.
Results
In two real-world problems, sLDA improved predictive performance over regression on unsupervised LDA features and lasso.
Takeaways & Limitations
The results support supervised dimension reduction when prediction is the ultimate goal.
Takeaways & Limitations
The formulation treats responses as generated from topic frequencies that actually occurred in the document, rather than from the mean topic-proportion distribution.
Abstract
from arXiv · showhide
We introduce supervised latent Dirichlet allocation (sLDA), a statistical model of labelled documents. The model accommodates a variety of response types. We derive an approximate maximum-likelihood procedure for parameter estimation, which relies on variational methods to handle intractable posterior expectations. Prediction problems motivate this research: we use the fitted model to predict response values for new documents. We test sLDA on two real-world problems: movie ratings predicted from reviews, and the political tone of amendments in the U.S. Senate based on the amendment text. We illustrate the benefits of sLDA versus modern regularized regression, as well as versus an unsupervised LDA analysis followed by a separate regression.
1. Introduction
The paper develops supervised topic models for documents with external response variables, targeting prediction rather than only descriptive topic discovery. sLDA is evaluated on movie ratings and Senate amendment tone, outperforming unsupervised LDA features with regression and lasso.
- Background: Topic models represent documents through smaller sets of latent semantic themes, supporting browsing, searching, and document-similarity assessment.A topic is formally a probability distribution over vocabulary terms.
- Motivation and contribution: Supervised topic models infer latent topics that predict an external response variable attached to each document.For an unlabeled document, the fitted model infers its topic structure and predicts the response.
- Motivation and contribution: Unsupervised LDA may emphasize dominant corpus structure, such as movie genres, instead of response-relevant distinctions such as evaluative language.For ratings prediction, useful topics should distinguish words such as “excellent,” “terrible,” and “average.”
- Motivation and contribution: sLDA supports general exponential-family responses and is demonstrated on movie-review ratings and the political tone of U.S. Senate amendments.The paper derives estimation and prediction algorithms, including specific procedures for Gaussian and Poisson responses.
- Motivation and contribution: In both applications, sLDA provides more predictive power than regression on unsupervised LDA features and improves on lasso.The experiments compare supervised topic modeling with both an unsupervised LDA-plus-regression pipeline and regularized regression.
2. Supervised latent Dirichlet allocation
sLDA extends LDA by jointly modeling document words and an associated response, using latent topic frequencies as predictors. A generalized linear-model response component accommodates multiple response types, while inference and estimation use variational approximations.
- LDA foundation: LDA represents each document with topic proportions drawn from a Dirichlet distribution and generates words through repeated topic assignments.Documents share K topics but use document-specific mixtures of those topics.
- sLDA model: sLDA adds a response variable to LDA and jointly models words and responses to find topics that predict responses for future unlabeled documents.Examples include movie ratings, article downloads, and document categories.
- Response model: The GLM response component uses an exponential dispersion family with natural parameter η⊤¯z and dispersion parameter δ.This framework includes normal, binomial, multinomial, Poisson, negative binomial, gamma, Weibull, and inverse Gaussian responses.
- sLDA model: The response is generated from a generalized linear model whose covariates are the document’s unobserved empirical topic frequencies.These frequencies constitute the observed topic assignments in the generated document, and the regression coefficients are η.
3. Computation with supervised LDA
sLDA requires approximate inference, parameter estimation, and prediction because the relevant posterior and likelihood quantities are intractable. The computation uses mean-field variational inference and variational EM, with response-aware updates for supervised topic modeling.
- Posterior inference: Posterior inference approximates the intractable conditional distribution of document-level topic proportions and topic assignments given words and model parameters.The approximation uses a parameterized variational distribution and optimizes its evidence lower bound.
- Parameter estimation: Variational EM estimates topics, Dirichlet hyperparameters, GLM coefficients, and dispersion by alternating document-level approximate posterior inference with corpus-level maximization.The procedure optimizes the corpus-level lower bound on log likelihood and finds a local optimum by iterating E- and M-steps.
- Prediction: For a new document, prediction approximates the posterior mean topic proportions using variational inference with response-dependent ELBO terms removed.The resulting topic representation is used to approximate the expected response under fixed model parameters.
- Posterior inference: The mean-field variational family fully factorizes topic proportions from per-word topic assignments to make optimization tractable.The variational parameters are a Dirichlet vector γ and categorical parameters φ1:N.
- Posterior inference: The sLDA ELBO differs from LDA through the expected log probability of the response given latent topic assignments.This response term requires expectations involving the GLM log-normalizer, which are exact for some response models and approximated in the general case.
- Posterior inference: Coordinate-ascent variational inference alternately updates γ and φ1:N, with the φn update incorporating the response-dependent ELBO contribution.Gaussian and Poisson responses permit exact coordinate updates, while other cases may require gradient-based optimization.
4. Empirical study
The empirical study evaluates sLDA on movie-review ratings and Senate-amendment political tone, comparing it with unsupervised LDA regression and lasso. sLDA improves prediction across the studied data, with the largest gains on Senate data.
- Datasets and prediction tasks: The study predicts movie ratings from reviews and amendment political tone from Senate texts using response variables derived from ratings and ideal-point analysis.The Senate response is each amendment’s issue-discrimination parameter.
- Evaluation: The evaluation uses five-fold cross-validation, measuring prediction quality by correlation and predictive R2.Predictive R2 is defined as the fraction of out-of-fold response variability captured by out-of-fold predictions.
- Baselines: sLDA is compared with regression on unsupervised LDA topic proportions and with lasso using empirical word distributions as covariates.The lasso comparison reports the best result across complexity settings, while sLDA uses the best result across topic counts.
- Results: sLDA provides improved predictions on all data across different numbers of topics, although movie ratings are easier to predict than Senate discrimination parameters.The results nevertheless show predictive power in amendment texts.
- Results: 0.426 versus 0.432: on movie reviews, the best lasso and sLDA predictive R2 values differ by a modest 2%.These are the best reported values for lasso and sLDA, respectively.
- Results: 0.15 versus 0.27: on 109th Senate data, sLDA improves predictive R2 over lasso by 80%; on 110th Senate data, 0.16 versus 0.23 yields a 43% improvement.The comparisons use the best lasso and sLDA predictive R2 values for each Senate dataset.
5. Discussion
The paper concludes that sLDA improves prediction over unsupervised LDA regression and lasso on two real-world problems. It also identifies semi-supervised extensions, added covariates, and applications to other LDA variants as future directions.
- Discussion: sLDA improved on unsupervised LDA followed by linear regression and on lasso in both real-world prediction problems.The authors present these results as benefits of supervised dimension reduction when prediction is the goal.
- Future directions: A semi-supervised sLDA can omit response terms for unlabeled documents, but the response data must exert sufficient influence on the fit.The paper identifies partially labelled corpora as a valuable setting for this extension.
- Future directions: Fixed-dimensional covariates can be added to the linear predictor, generally requiring an intercept term.The paper also proposes applying the response-integration technique to existing LDA variants.