PaperlayerView exact revision on arXiv

Source-linked AI summary

Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models

Mohammad Emtiyaz Khan, Wu Lin

arXiv:1703.04265v2cs.LG

TL;DR

Variational inference is challenging when models combine conjugate and non-conjugate terms, while stochastic-gradient methods may ignore conjugate structure and converge slowly. CVI combines stochastic gradients for non-conjugate terms with conjugate computations for the rest, and experiments show faster convergence than methods that ignore conjugacy.

  • Problem

    Variational inference in models with non-conjugate terms is computationally challenging, and stochastic-gradient methods may ignore conjugate structure, increasing parameterization and convergence difficulties.

  • Method

    CVI uses stochastic mirror descent in mean-parameter space, expressing gradient steps as variational inference in conjugate models while applying stochastic gradients only to non-conjugate terms.

  • Results

    Across many models and datasets, CVI converges much faster than methods that ignore conjugate structure.

  • Takeaways & Limitations

    CVI provides a general, modular, computationally efficient, scalable, and convergent approach that generalizes VMP and SVI to non-conjugate models.

  • Takeaways & Limitations

    CVI is motivated partly by limitations in related methods for arbitrary conjugacy structures and conditionally-conjugate inference requiring convergence before non-conjugate updates.

Abstract

from arXiv · show

Variational inference is computationally challenging in models that contain both conjugate and non-conjugate terms. Methods specifically designed for conjugate models, even though computationally efficient, find it difficult to deal with non-conjugate terms. On the other hand, stochastic-gradient methods can handle the non-conjugate terms but they usually ignore the conjugate structure of the model which might result in slow convergence. In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest. We derive this algorithm by using a stochastic mirror-descent method in the mean-parameter space, and then expressing each gradient step as a variational inference in a conjugate model. We demonstrate our algorithm's applicability to a large class of models and establish its convergence. Our experimental results show that our method converges much faster than the methods that ignore the conjugate structure of the model.

1 Introduction

Models combining conjugate and non-conjugate terms make variational inference computationally challenging. CVI combines stochastic gradients for non-conjugate terms with efficient conjugate computations and converges faster than methods that ignore conjugate structure.

  • Motivation: Variational inference remains computationally challenging for widely used models containing both conjugate and non-conjugate terms.Examples include Gaussian process classification, correlated topic models, probabilistic PCA, Kalman filters with non-Gaussian likelihoods, and deep exponential-family models.
  • Motivation: Conjugate computations are efficient, but non-conjugate terms disrupt their convenient exponential-family form and may require approximations that reduce performance.VMP relies on conjugate computations, while non-conjugate terms make its messages more complex.
  • Motivation: Stochastic-gradient methods handle broad inference problems but may depend on parameterization, use too many variational parameters, and converge slowly because they do not directly exploit conjugacy.These issues can make updates inefficient even when conjugate terms have closed-form structure.
  • Contribution: CVI uses stochastic gradients for non-conjugate terms while retaining conjugate computations for conjugate terms.Its stochastic mirror-descent formulation in mean-parameter space makes gradient steps implementable through conjugate computations.
  • Contribution: CVI reduces to VMP for conjugate models and converges much faster than methods that ignore conjugate structure.The paper also proves convergence and applies the method across many models and datasets.

2 Conjugate Computations

Conjugacy gives posterior distributions the same exponential-family form as the prior, allowing posterior natural parameters to be computed by adding prior and likelihood natural parameters.

  • Conjugacy: A conjugate prior has the same functional form as the likelihood, yielding a posterior available in closed form through simple computations.The paper calls these operations conjugate computations.
  • Exponential-family form: In an exponential family, the prior is represented using a natural parameter, sufficient statistics, a base measure, and a log-partition function.The natural parameter is updated through the likelihood's corresponding natural-parameter contribution.
  • Conjugate computation: For conjugate likelihoods, posterior inference adds the likelihood natural parameter to the prior natural parameter.This preserves the exponential-family form of the posterior.
  • Applications: Conjugate computations support Bayesian inference and variational-inference algorithms such as variational message passing and expectation propagation.Their computational efficiency motivates using them wherever model structure permits.

3 Non-Conjugate Variational Inference

Non-conjugate terms make variational lower-bound optimization difficult, while naive stochastic-gradient methods can ignore closed-form conjugate structure and use inefficient parameterizations.

  • Optimization challenge: Non-conjugate terms can make the variational lower-bound optimization intractable.For Gaussian-process models, expectations under non-Gaussian likelihoods commonly lack closed-form expressions.
  • Stochastic gradients: Stochastic-gradient descent offers a black-box solution for broad inference problems but may ignore conjugate terms that require no stochastic approximation.Its efficiency and convergence rate can also depend on the variational parameterization.
  • GP example: In GP models, a naive parameterization may use O(N^2) variational parameters even though only O(N) are free parameters.Choosing the best parameterization is difficult, and a naive choice can be substantially slower.
  • CVI response: CVI expresses each GP gradient step as predictions in a GP regression model with O(N) free parameters.The resulting updates avoid the full covariance computation and yield fast convergence in the reported results.

4 Conjugate-Computation Variational Inference (CVI)

CVI uses stochastic mirror descent in mean-parameter space to approximate non-conjugate terms while implementing each update with conjugate computation. The construction supports scalable inference across several model classes and converges under stated regularity conditions.

  • Core algorithm: CVI combines stochastic gradients for non-conjugate terms with conjugate computations for conjugate terms.The method is derived through stochastic mirror descent in mean-parameter space.
  • Assumptions: The method assumes a minimal exponential-family variational distribution and a model decomposition containing conjugate and non-conjugate parts.The conjugate part has the same form as the variational distribution with respect to the latent variables.
  • Conjugate implementation: Each mirror-descent update is equivalent to Bayesian inference in a conjugate model whose natural parameter is obtained by adding the approximated non-conjugate natural parameter to the conjugate parameter.The approximation is formed from a weighted sum of stochastic gradients.
  • GP implementation: For GP models, CVI updates are expressed as predictions in a GP regression model and require O(N) free parameters rather than the full covariance matrix.This produces substantial computational savings over naive SGD.
  • Extensions: The method extends to GLMs, Kalman filters with GLM likelihoods, Gamma variational distributions, and doubly stochastic updates.Doubly stochastic sampling reduces stochastic-gradient computations from O(N) to O(1) in the GP example.
  • Convergence: CVI converges to a local optimum when smoothness, unbiased bounded-variance stochastic approximation, and strong convexity conditions hold.The approach avoids explicitly forming the Fisher information matrix and instead solves a linear system.

5 CVI for Mean-Field Approximation

For mean-field models with conditional-conjugate structure, CVI combines conjugate message passing with stochastic gradients for non-conjugate terms, while retaining VMP- and SVI-like special cases and convergence guarantees.

  • CVI extends to conditionally-conjugate models by restricting the posterior approximation to a mean-field form.
  • The mean-field objective decomposes over nodes, enabling parallel optimization or doubly-stochastic updates that sample terms from the sum.
  • At each node, conjugate messages are computed by expected natural parameters, while stochastic-gradient messages handle the non-conjugate terms.
  • The resulting messages are combined into a natural parameter, followed by a convex combination to update the variational factor.
  • Without non-conjugate messages, CVI reduces to VMP, and with doubly-stochastic updates on conjugate-exponential models, it is equivalent to SVI.
  • The algorithm converges under both stochastic updates and parallel updates, including SVI-like and one-step VMP-like schemes.

6 Related Work

The paper positions CVI as a convergent stochastic alternative that retains conjugate structure, connecting it to non-conjugate VMP, proximal-gradient methods, and related stochastic optimization approaches.

  • Local variational approximations for non-conjugate terms need not converge to a lower-bound local maximum, whereas CVI's stochastic-gradient step is guaranteed to reach a stationary point.
  • Unlike EP's moment matching, CVI obtains its exponential-family approximation by maximizing the variational lower bound and provides convergence guarantees without EP's stated damping requirement.
  • NC-VMP is a special case of CVI when gradients are exact and the step-size is β_t = 1, making CVI a stochastic version with principled damping.
  • Related stochastic optimization can require explicitly storing a Fisher information matrix, which may be infeasible for models with many variational parameters such as Gaussian processes.
  • CVI uses mean-parameter mirror descent without splitting and yields updates corresponding to conjugate-model inference within a broader exponential-family class.

7 Results

CVI is evaluated on Bayesian logistic regression, gamma factor and matrix factorization, and GP classification, with results showing competitive or faster convergence across differing problem regimes.

  • CVI is evaluated on Bayesian logistic regression, gamma factor models, gamma matrix factorization, and Gaussian process classification.
  • Bayesian logistic regression: On covtype with N = 581K and D = 54, CVI is as fast as S&K-Alg2 and PG-exact, while Chol is slowest.
  • Bayesian logistic regression: On Colon-Cancer with N = 62 and D = 2000, CVI matches S&K-FG speed by directly estimating the mean gradient without forming the Fisher information matrix.
  • Bayesian logistic regression: CVI performs well in both N > D and D > N regimes because closed-form updates enable fast conjugate computations using the matrix-inversion lemma.
  • Gamma factor model: For the gamma factor model, CVI converges faster than the baseline while achieving the same perplexity error.
  • Poisson-gamma matrix factorization: For Poisson-gamma matrix factorization, CVI converges faster than the baseline and achieves a lower reconstruction error.

8 Conclusions

CVI combines stochastic gradients for non-conjugate terms with conjugate computations for the remaining terms. The paper presents it as a general, efficient, scalable, and convergent extension of VMP and SVI to non-conjugate models, while identifying settings where further improvements may be useful.

  • 8 Conclusions: CVI applies stochastic gradients only to non-conjugate parts while retaining conjugate computations for the rest.The method is based on stochastic mirror descent in mean-parameter space, with each gradient step expressible as variational inference in a conjugate model.
  • 8 Conclusions: CVI generalizes VMP and SVI to models containing both conjugate and non-conjugate terms.
  • 8 Conclusions: The method may simplify inference in deep generative models, but the cited related method does not easily generalize to arbitrary conjugacy structures.The authors also identify the need to run conditionally conjugate inference to convergence before updating non-conjugate parts as an issue their method may help address.

Supplementary Material For Conjugate-Computation Variational Inference

The supplementary material defines conjugacy as a relationship between a class of prior distributions and a class of data distributions. A prior class is conjugate when it is closed under the relevant posterior update.

  • Supplementary Material For Conjugate-Computation Variational Inference: Conjugacy is defined for a class of data distributions p(y|z) parameterized by z and a class of prior distributions for z.
  • Supplementary Material For Conjugate-Computation Variational Inference: The definition concerns whether the prior class P is conjugate for the data-distribution class F.
  • Supplementary Material For Conjugate-Computation Variational Inference: The supplementary material attributes this conjugacy definition to Gelman et al. (2014).

B Variational Inference in the GP Model and Issues with the SGD Algorithm

The GP example shows why naive SGD can be inefficient: Gaussian variational parameterizations can require large matrices and difficult inversions, despite the model having only O(N) free parameters. CVI addresses this by using conjugate computations and stochastic gradients for the non-conjugate terms.

  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: For Gaussian q, the KL term and its gradients have closed-form expressions, so stochastic-gradient approximations are unnecessary for that term.The expected non-conjugate likelihood term may remain intractable, whereas the conjugate contribution can be computed directly.
  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: Different Gaussian parameterizations produce computationally different updates, including updates involving V^-1 or V.The parameterizations include natural parameters, mean parameters, and {m, V}.
  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: Both inverse-covariance and covariance updates may require explicitly forming V and V^-1, and both compute an inverse of K that might be ill-conditioned.These operations may be infeasible for large N, such as a million data points.
  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: The discussed GP parameterization requires O(N^2) memory even though the model has only O(N) free parameters.Choosing any of the three earlier parameterizations is described as an order of magnitude slower than the best option.
  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: CVI avoids the re-parameterization issue by expressing gradient steps as conjugate computations and using stochastic gradients only for non-conjugate terms.Its updates naturally have O(N) free variational parameters, and a doubly-stochastic scheme can reduce gradients per iteration to O(1).
  • B Variational Inference in the GP Model and Issues with the SGD Algorithm: The derivation converts mean-parameter gradients into a conjugate-model update through Bregman and KL-divergence identities.The resulting distribution is represented as the posterior of a conjugate model, with recursion expressing successive updates in that form.

H Dataset Details

The dataset-details section identifies the sources and organization of the experimental datasets and records the model and algorithmic settings used in evaluation. The experiments include comparisons with several alternative inference methods.

  • H Dataset Details: The experiments use datasets for Bayesian logistic regression, gamma factor models, and Gaussian-process classification obtained from specified public sources.
  • H Dataset Details: Model hyperparameters are tuned by grid search and then fixed during the experiments, while Table 2 reports algorithmic and model parameters.
  • H Dataset Details: Table 1 lists models and datasets, including NTrain, K, and hyperparameter values.For GP classification, σf and l are hyperparameters of the squared-exponential kernel.
  • H Dataset Details: The study compares CVI with PG-SVI, CVI-exact, and expectation propagation for GP classification.The methods are introduced as additional comparison methods in the experiments.

I.1 Additional Results

Additional experiments compare CVI with established methods across Bayesian logistic regression, Gaussian process classification, gamma factor models, and gamma matrix factorization. The reported results emphasize comparable predictive performance and faster convergence for CVI in several settings.

  • Bayesian logistic regression: CVI is compared with multiple methods for Bayesian logistic regression on seven real datasets, with all methods reaching the same performance.The comparison includes convergence behavior and performance across regimes where the number of examples and dimensionality differ.
  • Gaussian process classification: For Gaussian process classification, the experiments compare CVI, CVI-exact, expectation propagation, Chol, and PG-SVI.CVI and PG-SVI use Monte Carlo gradient approximations, while CVI-exact uses exact gradients.
  • Model specifications: The gamma factor and gamma matrix factorization experiments use Gamma-distributed variational factors or posteriors for their respective models.The gamma factor model uses a mean-field approximation with Gamma factors, while gamma matrix factorization uses a Gamma posterior.
  • Gamma factor model: The gamma factor model results report that CVI is much faster than the Knowles method.Table 4 summarizes the comparison, with lower values indicating better performance.
  • Gamma matrix factorization: The gamma matrix factorization results report that CVI outperforms ADAM.Table 5 presents the comparison for this model.
Loading 1703.04265v2…