Source-linked AI summary
On Estimation and Selection for Topic Models
Matthew A. Taddy
TL;DR
Topic-model estimation and selection face computational burdens, approximation concerns, and limited guidance for choosing the number of topics. The paper uses joint MAP estimation under a natural exponential-family parametrization, then applies block-diagonal Laplace approximation and residual dispersion for model assessment. The method reportedly matches or outperforms alternatives while scaling to large text datasets, although variational independence assumptions and dispersion-based fit assessment have limitations.
Problem
Existing topic-model estimation can be computationally costly and potentially biased, while the literature lacks a default method for selecting the number of topics.
Method
The paper jointly maximizes the posterior under a natural exponential-family parametrization and estimates marginal likelihood using a block-diagonal Laplace approximation, complemented by residual-dispersion analysis.
Results
Joint MAP estimation uniformly met or outperformed alternative estimates and fit 20 or more topics to hundreds of thousands of documents and tens of thousands of terms in less than ten minutes on a common desktop.
Takeaways & Limitations
The marginal-likelihood approximation supports efficient data-driven topic selection, while residual dispersion can measure goodness of fit but is complicated by bias and alternative overdispersion sources.
Takeaways & Limitations
Full variational Bayes risks inconsistency because cross-topic posterior correlations do not disappear asymptotically, and residual-dispersion interpretation is complicated by bias and alternative overdispersion.
Abstract
from arXiv · showhide
This article describes posterior maximization for topic models, identifying computational and conceptual gains from inference under a non-standard parametrization. We then show that fitted parameters can be used as the basis for a novel approach to marginal likelihood estimation, via block-diagonal approximation to the information matrix,that facilitates choosing the number of latent topics. This likelihood-based model selection is complemented with a goodness-of-fit analysis built around estimated residual dispersion. Examples are provided to illustrate model selection as well as to compare our estimation against standard alternative techniques.
1 Introduction
Topic models represent count data using mixtures of latent topics, while existing estimation and topic-selection methods leave computational, bias, and methodological gaps. The article addresses these gaps with an efficient estimation alternative and default model-selection methods.
- Topic models represent document term counts through mixed-membership weights over probability distributions for latent topics.With K far smaller than p, each document receives a reduced-dimension summary through its topic weights.
- Large-data estimation commonly maximizes an approximation to marginal likelihood using high-dimensional latent-variable augmentation.This augmentation increases computational cost and relies on independence assumptions that may bias estimation.
- The literature offers limited methodology for selecting the number of latent topics K.
- The article develops joint posterior maximization for estimation and two model-choice metrics based on marginal likelihood and residual dispersion.The proposed methods are illustrated with simulations and data examples.
2 Background
Topic models developed from mixed-membership likelihoods into Bayesian latent Dirichlet allocation and related extensions. Existing estimation methods use posterior approximations or latent-variable sampling, while topic selection relies on tools with important computational or interpretive limitations.
- Bayesian topic modeling commonly uses latent Dirichlet allocation, while related extensions address topic number, temporal change, correlation, and sentiment.
- Posterior approximations augment the model with term-level topic memberships, creating a high-dimensional latent parameter matrix Z.
- Variational estimation maximizes a lower bound on marginal likelihood under an independence approximation for latent memberships.
- Full variational Bayes assumes independent topic distributions even though cross-topic posterior correlations do not disappear asymptotically.The cited discussion identifies risks of inconsistency and unstable finite-sample estimates.
- Joint MAP estimation instead optimizes topics and weights together, avoiding posterior approximation but connecting more closely to maximum-likelihood EM methods.
- Cross-validation is common for topic selection but requires repeated model fitting and lacks straightforward interpretation as statistical evidence or sample error.
3 Parameter Estimation
The estimation procedure transforms topic and document-weight parameters into natural exponential family coordinates and performs joint MAP optimization. This parametrization yields concave conditional posteriors, enabling stable, parallel updates within a block-relaxation algorithm.
- 3.1 Natural Exponential Family Parameterization: The method estimates topics and document weights in natural exponential family parameterization rather than the original simplex space.Document-weight parameters become unrestricted vectors of length K − 1 after fixing one element.
- 3.1 Natural Exponential Family Parameterization: The NEF parameterization removes the “-1 offset” in MAP estimation and aligns the resulting estimates with simplex MAPs under shifted Dirichlet priors.
- 3.1 Natural Exponential Family Parameterization: The transformed conditional posterior is everywhere concave, guaranteeing a single conditional MAP solution for each document weight vector.This also produces non-boundary estimates that facilitate Laplace approximation.
- 3.2 Joint Posterior Maximization: Joint MAP estimation uses EM calculations based on latent topic totals rather than the full set of term-level memberships.
- 3.2 Joint Posterior Maximization: Exact conditional solutions for document weights replace slow EM updates and can accelerate convergence through independent parallel optimization.The procedure uses sequential quadratic programming and quasi-Newton acceleration.
- 3.2 Joint Posterior Maximization: The active-set strategy enforces positivity and the equality constraint on document weights while solving the quadratic-programming update.
- 3.2 Joint Posterior Maximization: Conditional log-posterior concavity guarantees a unique solution at each iteration, although the full transformed joint posterior remains multimodal.The algorithm therefore uses initialization intended to avoid minor modes.
4 Model Selection
The paper proposes two computationally light methods for selecting the number of topics: block-diagonal marginal-likelihood approximation and residual-dispersion analysis. The first approximates an otherwise intractable integral efficiently, while the second tests whether fitted overdispersion suggests too few topics.
- Model selection: The section proposes marginal likelihood approximation and residual analysis as two techniques for inferring the number of topics.Both require almost no computation beyond parameter estimation.
- Marginal likelihood maximization: Marginal likelihood model selection is motivated by an intractable integral over topic and weight parameters.Laplace’s method replaces the joint parameter-data likelihood near its MAP with an integrated Gaussian approximation.
- Marginal likelihood maximization: The non-standard exponential-family parameterization avoids boundary solutions where Laplace’s approximation would be invalid and lowers the integral’s dimension.The topic parameters remain in simplex form because their transformed Hessian is denser and harder to approximate.
- Marginal likelihood maximization: The full Hessian determinant is computationally burdensome for large text datasets, motivating a block-diagonal factorization.The approximation ignores sparse off-diagonal curvature terms and replaces the determinant with a product of diagonal-block determinants.
- Marginal likelihood maximization: The block-diagonal marginal likelihood estimate is fast, easy to calculate, and performs well in finite-sample examples.Its off-diagonal terms primarily represent covariance between shared topic vectors and individual document weights.
- Residuals and dispersion: Residual dispersion compares fitted phrase counts against multinomial sampling variance, with overdispersion ˆσ2 > 1 indicating that the true topic count may exceed the fitted count.The estimate uses adjusted squared residuals and tests D against an approximate χ2_ν distribution under σ2 = 1.
5 Examples
The examples assess model selection, estimator accuracy and computation, predictive performance, and topic interpretation using simulations and two text datasets. MAP generally performs favorably against VEM and Gibbs, while likelihood and dispersion support topic-number selection but reveal additional overdispersion in real data.
- Simulation study: In simulations, marginal likelihood selected the true K = 10 model invariably for M ≥200, while K = 8 was most common at M = 100.For M = 100, the likelihood was relatively flat before dropping for K > 10.
- Simulation study: Estimated dispersion was almost always above one for K < 10 and below one for K ≥10, with separation increasing as expected document size M increased.The estimates were biased low by roughly 1-6%, and the χ2 test gave p = 0 for K < 10 and p = 1 for K ≥10.
- Simulation study: MAP and VEM both estimated simulated topics largely accurately, but MAP had slightly lower MSE while VEM took at least 2-3 times longer to converge.Gibbs had far larger MSE than either alternative, consistent with estimation from a single draw after a short MCMC run.
- Data analysis: For real datasets, marginal likelihood was maximized at K = 20 for we8there and K = 12 for congress109.Dispersion remained above one even at the selected models and did not approach one for K up to 200.
- Data analysis: MAP produced higher predictive probability than VEM for we8there and nearly identical performance for congress109, while VEM convergence took many times longer.Gibbs performed well relative to VEM but worse than MAP estimators.
- Data analysis: The fitted models reduced dimensionality from 1000 to 12 and from 2804 to 20 by replacing individual phrases with topic weights.Congress109 topics were summarized using high-lift terms, usage proportions, and party-linked topic weights; some topics were strongly partisan.
6 Discussion
The methodology receives general support: marginal likelihood appears to enable efficient, data-driven selection of the number of topics, while joint MAP estimation maintains or improves fit and scales to large datasets. Residual dispersion may assess goodness of fit, but bias and other overdispersion sources complicate its interpretation.
- Marginal likelihood approximation appears to provide efficient, data-driven selection of the number of latent topics.The paper presents this as a default approach where the literature had lacked one.
- Residual dispersion shows potential for goodness-of-fit measurement, but bias and alternative sources of overdispersion complicate its role.
- Joint MAP estimation uniformly met or outperformed alternative estimates without lower-quality fit.
- Parallel independent updates for each ω_i|Θ allow fitting 20 or more topics to hundreds of thousands of documents in less than ten minutes on a common desktop.