PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Inference in Nonconjugate Models

Chong Wang, David M. Blei

arXiv:1209.4360v4stat.ML

TL;DR

Many useful probabilistic models are nonconjugate, preventing the standard closed-form mean-field coordinate updates and forcing model-specific variational algorithms. This paper develops Laplace and delta method variational inference for a broad nonconjugate class, and finds better results than special-purpose methods across three models, with Laplace usually outperforming delta method inference.

  • Problem

    Nonconjugate models lack the properties needed for standard closed-form mean-field coordinate updates, requiring variational algorithms tailored to each model.

  • Method

    The paper develops Laplace variational inference and delta method variational inference, which approximate the variational objective within mean-field optimization for a large class of nonconjugate models.

  • Results

    Across Bayesian logistic regression, hierarchical logistic regression, and the correlated topic model, the methods give better results than special-purpose techniques, while Laplace usually outperforms delta method inference in computation time and posterior fidelity.

  • Takeaways & Limitations

    The methods expand mean-field variational inference to models with fewer restrictions than conditional conjugacy and produce approximate posteriors that lead to good predictions.

  • Takeaways & Limitations

    The corresponding objective function for the approximate coordinate-ascent approach is not clearly characterized and remains an open problem.

Abstract

from arXiv · show

Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the model is conditionally conjugate, the coordinate updates are easily derived and in closed form. However, many models of interest---like the correlated topic model and Bayesian logistic regression---are nonconjuate. In these models, mean-field methods cannot be directly applied and practitioners have had to develop variational algorithms on a case-by-case basis. In this paper, we develop two generic methods for nonconjugate models, Laplace variational inference and delta method variational inference. Our methods have several advantages: they allow for easily derived variational algorithms with a wide class of nonconjugate models; they extend and unify some of the existing algorithms that have been derived for specific models; and they work well on real-world datasets. We studied our methods on the correlated topic model, Bayesian logistic regression, and hierarchical Bayesian logistic regression.

1 Introduction

Mean-field variational inference is efficient and broadly useful, but its standard closed-form coordinate updates depend on conditional conjugacy. The paper develops generic Laplace and delta-method approaches for applying mean-field inference to a wider class of nonconjugate models.

  • Mean-field inference fits independent latent-variable distributions by minimizing KL divergence to the posterior, turning inference into optimization.
  • Conditionally conjugate models support easily derived closed-form coordinate-ascent updates for variational parameters.
  • Nonconjugate models lack the required properties for these updates, so variational inference traditionally requires model-specific algorithms.
  • The paper develops Laplace variational inference by embedding Laplace approximations in variational optimization and delta method variational inference by optimizing a Taylor approximation.
  • The proposed methods require fewer restrictions than conditional conjugacy and significantly expand the models to which mean-field inference can be easily applied.
  • Across Bayesian logistic, hierarchical logistic, and correlated topic models, the methods outperform special-purpose techniques; Laplace usually improves computation time and posterior fidelity over delta method inference.

2 Variational Inference and a Class of Nonconjugate Models

Mean-field variational inference approximates difficult posteriors with a factorized family and is especially tractable under conditional conjugacy. The paper defines a broader nonconjugate class in which one variable is nonconjugate while the remaining conditional structure stays tractable.

  • Variational inference approximates the posterior p(θ, z|x) with a simpler distribution q(θ, z) by minimizing KL divergence.
  • Mean-field inference uses a fully factorized family whose variables are independent, which generally cannot represent posterior dependence but can capture latent-variable marginals.
  • Minimizing KL divergence is equivalent to maximizing a Jensen-derived lower bound on the observed-data log marginal likelihood, called the variational objective.
  • Coordinate ascent alternately updates q(θ) and q(z), converging to a local optimum; conditional conjugacy makes these updates available in closed form.
  • If any model node is nonconjugate, closed-form coordinate ascent and easy objective calculation are unavailable, motivating generic algorithms.
  • The broader class assumes twice-differentiable p(θ), an exponential-family p(z|θ) with twice-differentiable natural parameter, and an exponential-family p(x|z).
  • Unlike conjugate models, these assumptions do not require p(θ) and p(z|θ) to form a conjugate pair, so p(θ|z) need not match the prior family.
  • The class includes correlated and dynamic topic models, Bayesian logistic and discrete choice models, and Bayesian ideal point models, with adaptations to more complex graphical models.

3 Laplace and Delta Method Variational Inference

The paper develops Laplace and delta method variational inference algorithms for nonconjugate models, where exact mean-field updates are difficult. Both use coordinate ascent, differing in how they update the nonconjugate variational distribution.

  • Motivation: Nonconjugate models make mean-field updates difficult because the relevant update may not have a usable exponential-family form or closed-form expectations.The paper targets models where p(θ) is not conjugate to p(z | θ), preventing direct application of standard coordinate updates.
  • Shared algorithm: Both methods use mean-field coordinate ascent, alternating updates of the nonconjugate q(θ) and conjugate q(z) distributions.The conjugate variable remains in the same family as p(z | η(θ)), while q(θ) is Gaussian in both variants.
  • Laplace variational inference: Laplace variational inference approximates the nonconjugate update by a Gaussian constructed from a second-order Taylor expansion around the maximum of f(θ).The maximizer is found numerically, and the Gaussian form arises from the Taylor approximation rather than an initial Gaussian assumption.
  • Delta method variational inference: Delta method variational inference optimizes a Taylor approximation of the variational objective while fitting Gaussian parameters for q(θ).Different Taylor constructions yield different algorithms; one recovers the Laplace approximation, while another uses a multivariate delta approximation.
  • Algorithmic properties: Deriving updates becomes mechanical work involving model-specific derivatives and numerical optimization, with Laplace inference requiring second rather than third derivatives.The approximate variational objective increased over iterations in typical correlated topic model runs.

4 Example Models

The paper applies its generic nonconjugate variational-inference algorithm to correlated topic, Bayesian logistic, and hierarchical logistic regression models. These examples show how the models’ nonconjugate and conjugate variables are identified and how inference integrates with model fitting.

  • The generic algorithm is derived for the correlated topic model, Bayesian logistic regression, and hierarchical Bayesian logistic regression.
  • Correlated Topic Model: The correlated topic model replaces LDA’s Dirichlet prior with a logistic normal prior that can capture correlations between topic proportions.Its latent topic assignments generate observed words from topic-specific distributions.
  • Correlated Topic Model: CTM inference computes p(θ, z1:N | x1:N, β1:K) for prediction and as a subroutine of variational expectation maximization for fitting topics and logistic-normal parameters.
  • Correlated Topic Model: In the correlated topic model, topic proportions θ have a Gaussian variational distribution, while topic assignments z have a discrete variational distribution.Delta method inference restricts the covariance Σ to be diagonal, whereas Laplace variational inference does not require this simplification.
  • Bayesian Logistic Regression: In Bayesian logistic regression, the Gaussian prior on coefficients is nonconjugate to the label likelihood, and Laplace variational inference recovers the standard Laplace approximation.The hierarchical extension applies the inference procedure separately to related problems and estimates shared prior hyperparameters through variational EM.
  • The same algorithms can also be adapted to other Bayesian generalized linear models and several nonconjugate topic models.

5 Empirical Study

The empirical study evaluates the proposed nonconjugate inference methods on correlated topic models and Bayesian logistic regression. Across these settings, Laplace inference generally provides the strongest combination of predictive performance, accuracy, fidelity, and computational speed.

  • Experimental setup: The study evaluates Laplace and delta method inference against model-specific approaches on correlated topic models and Bayesian logistic regression.Experiments use variational EM for topic models and classification evaluations for logistic regression.
  • Correlated topic model: Laplace inference converges faster and achieves better predictive performance than delta method inference during variational EM.Figure 6 compares the approximate bound and held-out likelihood against running time with K = 60 topics.
  • Correlated topic model: On correlated topic models, both nonconjugate methods outperform Blei and Lafferty’s tailored approach on held-out likelihood, while Laplace inference performs best.Blei and Lafferty’s objective is erratic across collections and may be a loose bound despite being a strict lower bound.
  • Bayesian logistic regression: On Bayesian logistic regression, Laplace and delta method inference give slightly better accuracy and much better log predictive likelihood than Jaakkola and Jordan’s method.A t-test at level 0.05 found both nonconjugate methods better than the model-specific baseline.
  • Bayesian logistic regression: In hierarchical logistic regression, hierarchical models outperform non-hierarchical models in both accuracy and predictive likelihood, with hierarchical Laplace inference significantly best on predictive likelihood.The comparison uses a standard t-test at level 0.05.

6 Discussion

The paper finds that Laplace and delta method variational inference work well across nonconjugate models, with Laplace inference better and faster than delta inference in the analyzed examples.

  • 6 Discussion: Laplace and delta method inference produced good predictions and performed better than methods tailored to the specific models in the analyzed examples.The methods were studied in two nonconjugate models and formed approximate posteriors that led to good predictions.
  • 6 Discussion: Standard Bayesian logistic regression algorithms performed the same as algorithms developed for Yeast and Scene as multi-task problems.
  • 6 Discussion: Laplace and delta method inference performed best for Bayesian logistic regression on accuracy and averaged log predictive likelihood.The table results were averaged from five random starts, with higher numbers better.
  • 6 Discussion: The hierarchical model performed best on the School data across separate, pooled, and hierarchical logistic regression settings.Results were averaged from 10 random splits, and significance was assessed with a standard t-test at level 0.05.
  • 6 Discussion: Laplace inference was better and faster than delta method inference.

Appendix A: Generalization to Complex Models

The appendix generalizes mean-field coordinate-ascent inference to complex models by retaining exact updates for conjugate variables and approximating otherwise intractable factors with Laplace or delta methods.

  • Appendix A: Generalization to Complex Models: The goal is to approximate p(θ | x) with a fully factorized mean-field variational family and optimize each factor by KL divergence.
  • Appendix A: Generalization to Complex Models: Coordinate ascent iteratively optimizes each variational factor while holding the others fixed.
  • Appendix A: Generalization to Complex Models: The general coordinate update has the form q(θi) ∝ exp{f(θi)} after terms constant in θi are absorbed into the proportionality constant.
  • Appendix A: Generalization to Complex Models: Laplace variational inference approximates f(θi) around its maximum ˆθi, yielding a Gaussian factor when the update is otherwise intractable.
  • Appendix A: Generalization to Complex Models: In the general coordinate-ascent setting, expected log probabilities and their derivatives must be computed or approximated.
  • Appendix A: Generalization to Complex Models: The framework applies when factors belong to the exponential family, a weaker condition than conjugacy that describes most graphical models from the literature.
  • Appendix A: Generalization to Complex Models: Further Taylor approximations can be applied to natural parameters, sufficient statistics, and log normalizers to compute their expectations.
  • Appendix A: Generalization to Complex Models: Some conjugate variables can be updated exactly because their complete conditional distributions are available in closed form.

Appendix B: The Correlated Topic Model

For the correlated topic model, the appendix derives the quantities needed by Laplace and delta variational inference and uses them in posterior inference within variational EM.

  • Appendix B: The Correlated Topic Model: The correlated topic model implementation identifies the quantities from earlier equations needed to compute f(θ).
  • Appendix B: The Correlated Topic Model: Eq(z) [t(z)] represents expected word counts for each topic under the variational distribution q(z).
  • Appendix B: The Correlated Topic Model: Using the topic-proportion transformation π = η(θ), the appendix derives the gradient and Hessian of f(θ) for the correlated topic model.
  • Appendix B: The Correlated Topic Model: Only the gradient of f(θ) is needed for Laplace inference in the correlated topic model.
  • Appendix B: The Correlated Topic Model: Delta method variational inference additionally requires the gradient of the trace term and assumes that Σ is diagonal.
  • Appendix B: The Correlated Topic Model: The derived quantities implement an algorithm for inferring per-document posteriors of the correlated topic model's hidden variables.
  • Appendix B: The Correlated Topic Model: Within variational EM, the E-step performs posterior inference for each document, while the M-step follows the variational family described by Blei and Lafferty.

Appendix C: Bayesian Logistic Regression

The appendix applies the variational methods to Bayesian logistic regression and describes global hyperparameter updates for the hierarchical model.

  • Appendix C: Bayesian Logistic Regression: The Bayesian logistic regression observations are represented in the exponential family, with observed labels serving as their own expectations.
  • Appendix C: Bayesian Logistic Regression: The resulting f(θ) supports the standard Laplace approximation to Bayesian logistic regression.
  • Appendix C: Bayesian Logistic Regression: Delta variational inference requires the gradient of the trace term, but the Hessian is already diagonal, so Σ need not be assumed diagonal.
  • Appendix C: Bayesian Logistic Regression: In hierarchical logistic regression, the variational distribution of each problem's coefficients θm is computed before estimating global hyperparameters (µ0, Σ0) by MAP.
  • Appendix C: Bayesian Logistic Regression: The coefficient dimension is denoted by p.
Loading 1209.4360v4…