Source-linked AI summary
Nonparametric variational inference
Samuel Gershman, Matt Hoffman, David Blei
TL;DR
Variational inference is limited by restricted families and conjugacy requirements, especially when posteriors are multimodal or models are non-conjugate. The paper introduces NPV, a Gaussian-mixture approximation optimized through an approximate ELBO, and demonstrates performance comparable to specialized variational inference and better than several sample-based methods.
Problem
Traditional variational methods often use unimodal families and are difficult to derive for non-conjugate models.
Method
NPV uses a Gaussian-mixture variational family with optimized component locations and variances, approximating the objective with Taylor expansions and an entropy bound.
Results
NPV performs as well as a specialized variational method in Bayesian logistic regression and outperforms several MCMC methods for non-conjugate matrix factorization.
Takeaways & Limitations
NPV is presented as a generally applicable variational approximation for complex probabilistic models, requiring computable derivatives of the log joint probability.
Takeaways & Limitations
NPV assumes a simple approximating family and currently applies only to continuous hidden variables.
Abstract
from arXiv · showhide
Variational methods are widely used for approximate posterior inference. However, their use is typically limited to families of distributions that enjoy particular conjugacy properties. To circumvent this limitation, we propose a family of variational approximations inspired by nonparametric kernel density estimation. The locations of these kernels and their bandwidth are treated as variational parameters and optimized to improve an approximate lower bound on the marginal likelihood of the data. Using multiple kernels allows the approximation to capture multiple modes of the posterior, unlike most other variational approximations. We demonstrate the efficacy of the nonparametric approximation with a hierarchical logistic regression model and a nonlinear matrix factorization model. We obtain predictive performance as good as or better than more specialized variational methods and sample-based approximations. The method is easy to apply to more general graphical models for which standard variational methods are difficult to derive.
1. Introduction
Traditional variational inference often uses restricted, unimodal families and is difficult to derive for non-conjugate models. NPV uses a Gaussian-mixture family inspired by kernel density estimation to capture multimodality and apply more broadly.
- Background: Variational inference approximates the posterior by optimizing within a tractable restricted family of distributions.The family is chosen to make the optimization problem tractable.
- Limitations of traditional methods: Mean-field variational methods usually produce unimodal approximations, which are inadequate for multimodal posteriors.They are also difficult to derive for models lacking conditional conjugacy.
- Nonparametric variational inference: NPV uses a mixture of Gaussians whose component locations and variances are optimized as variational parameters.The method approximates its objective using Taylor expansions of the log joint distribution and an entropy bound.
- Nonparametric variational inference: The Gaussian-mixture family can capture different posterior aspects and is inspired by classical nonparametric kernel density estimators.Multiple mixture components support multimodal approximations unlike traditional unimodal variational distributions.
- Generality: NPV requires computable first and second derivatives of the log joint probability rather than model-specific conjugacy mathematics.The paper presents it as adaptable to many non-conjugate settings.
- Empirical demonstrations: In Bayesian logistic regression NPV performs as well as a specialized variational method, while in non-conjugate matrix factorization it outperforms several MCMC methods.These demonstrations motivate NPV as a strategy for complex probabilistic models.
2. Variational inference
Variational inference replaces an analytically intractable posterior computation with optimization over a tractable family. The ELBO provides the optimization objective, while mean-field factorization offers convenience but can be restrictive and depends on conjugacy for closed-form updates.
- Posterior inference: Posterior computation can be analytically intractable because the normalization denominator is difficult to compute.The target is the posterior distribution of hidden variables given observed data.
- Variational objective: Variational inference approximates the posterior with a constrained family and chooses its member by minimizing KL divergence.The family is selected to make this optimization tractable.
- Variational objective: The ELBO is a lower bound on log p(y) and equals the negative KL divergence plus a constant with respect to the variational family.It reaches its maximum when the variational distribution equals the true posterior, when that posterior lies in the family.
- Mean-field inference: Mean-field variational inference uses factorized posteriors and can obtain closed-form coordinate updates when priors are conjugate to the likelihood.These updates converge to a local maximum of the variational objective.
- Limitations of mean-field inference: Mean-field approximations can be overly restrictive when posterior hidden variables have strong dependencies.Closed-form updates are unavailable for many non-conjugate likelihood models, including logistic regression and multilayer perceptrons.
3. Nonparametric variational inference
NPV uses a Gaussian-mixture variational family to represent multimodal posteriors without conjugacy, optimizing an approximate ELBO with tractable derivative requirements. Its flexibility comes with computational and representational trade-offs as component count grows.
- 3. Nonparametric variational inference: NPV represents continuous hidden variables with a uniformly weighted Gaussian mixture whose component means and variances are variational parameters.The mixture resembles a kernel density estimator, with means as kernel centers and variances as bandwidth parameters.
- 3.1. The Evidence Lower Bound: The approximate ELBO combines a Jensen entropy bound with a second-order Taylor approximation of the expected log joint.The likelihood term favors high-density locations, the entropy term discourages overcrowding, and the Hessian term accounts for local curvature.
- 3.1. The Evidence Lower Bound: The objective requires no conjugacy and only diagonal Hessian components, whose computation costs about as much as computing the gradient.The log joint must be twice differentiable, or thrice differentiable when using gradient ascent.
- 3.2. Optimizing the ELBO: The optimization alternates second-order variance updates with first-order mean updates to avoid computing third derivatives.Means are optimized with L-BFGS, and coordinate ascent over components can converge faster than batch optimization.
- 3.2. Optimizing the ELBO: With N = 2, NPV captures two modes with high fidelity, but isotropic components miss the true covariance structure and heavy tails.Using N = 10 improves skew representation through several low-variance components along the diagonal.
- 3.2. Optimizing the ELBO: The number of fitted parameters grows linearly with N, while KL divergence improves at best logarithmically, implying diminishing returns from many components.A small number of components may capture major posterior features, but large hidden-variable models may pose fitting challenges.
- 3.3. Relationship to other algorithms: When N = 1 and variance approaches zero, NPV reduces to a MAP objective; varying the variance yields a diagonalized Laplace approximation.As component variances approach zero, the method can also be viewed as a deterministic sampling approximation.
4. Related work
Existing approaches to non-conjugate inference include numerical, Monte Carlo, and model-specific approximations. NPV instead targets general non-conjugate models with continuous hidden variables while relating to mixture mean-field methods.
- Numerical and Monte Carlo methods approximate intractable integrals but may fail in high dimensions.
- Specialized approximations address particular model classes, whereas NPV targets general non-conjugate models with continuous hidden variables.
- NPV is a special case of mixture mean-field because each mixture component factorizes into one-dimensional Gaussian sub-components.
5. Applications
The applications evaluate NPV on hierarchical logistic regression and nonlinear fMRI matrix factorization. NPV matches a specialized variational method in logistic regression and improves predictive performance over MAP and MCMC estimators for TLSA.
- 5.1. Logistic regression: NPV and JJ achieve statistically indistinguishable test-data accuracy and closely matching ELBO values on 13 binary classification data sets.
- 5.1. Logistic regression: NPV uses only derivatives of the joint distribution, whereas JJ exploits a model-specific lower bound on the logistic sigmoid.
- 5.1. Logistic regression: With 1000 samples, HMC predicted held-out logistic-regression data significantly worse than NPV and JJ (p < 0.00001).
- 5.2. Topographic latent source analysis: TLSA decomposes voxel activations into covariate-dependent combinations of spatial latent sources and predicts held-out brain data conditional on covariates.
- 5.2. Topographic latent source analysis: Each NPV component captures idiosyncrasies corresponding to different local optima that the MAP reconstruction misses.
- 5.2. Topographic latent source analysis: NPV substantially outperformed MAP and MCMC estimators on held-out neural-data reconstruction, performing best with more than three components.
- 5.2. Topographic latent source analysis: NPV was about 3 times faster than HMC in the TLSA experiments.
6. Discussion
NPV is designed for posterior inference without conjugacy, with straightforward model application and competitive predictive performance. Its current scope is limited by a simple approximating family and continuous hidden variables.
- Discussion: NPV requires only the likelihood function and its gradient, making it easy to apply to new probabilistic models without conjugacy.This requirement is shared by methods including MAP estimation and HMC.
- Discussion: In hierarchical logistic regression, NPV incurs little loss in accuracy compared with a specialized variational algorithm.
- Discussion: In a nonlinear latent variable model of fMRI data, NPV improves predictive performance over MAP estimation and MCMC.
- Discussion: NPV assumes a simple approximating family, which could be expanded using full covariance matrices or nonuniform component weights.
- Discussion: NPV currently applies only to continuous variables, with extensions to discrete hidden variables planned.
- Discussion: NPV is presented as a step toward generically applicable variational approximations for increasingly complex probabilistic models.