Source-linked AI summary
Stochastic gradient variational Bayes for gamma approximating distributions
David A. Knowles
TL;DR
Bayesian inference remains challenging to implement and scale, and gradient-estimate variance complicates Monte Carlo variational inference. The paper applies log-posterior-gradient variational inference to gamma latent variables using gamma variational distributions. It reports that gamma SGVB improves over the IRM on network data, while carefully engineered MCMC performs best there.
Problem
Bayesian inference is difficult to implement and scale, and Monte Carlo variational inference must control gradient-estimate variance; prior log-posterior-gradient results focused on Gaussian latent variables.
Source: §1 · pp. 1–2Method
The paper applies stochastic gradient variational Bayes to gamma latent variables with gamma variational distributions, using Monte Carlo gradients based on the log posterior.
Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 4Results
On network data, gammaSGVB improves over the IRM, although the carefully engineered MCMC algorithm consistently performs best.
Source: §4.1 · p. 9Takeaways & Limitations
The method applies log-joint-gradient variational inference to sparse continuous gamma latent variables, with the model-specific requirement being the log-joint gradient.
Source: §5 · p. 12The gamma shape-parameter gradient has no analytical form in commonly available special functions, so it requires approximation.
Source: §2.3 · p. 4
Abstract
from arXiv · showhide
While stochastic variational inference is relatively well known for scaling inference in Bayesian probabilistic models, related methods also offer ways to circumnavigate the approximation of analytically intractable expectations. The key challenge in either setting is controlling the variance of gradient estimates: recent work has shown that for continuous latent variables, particularly multivariate Gaussians, this can be achieved by using the gradient of the log posterior. In this paper we apply the same idea to gamma distributed latent variables given gamma variational distributions, enabling straightforward "black box" variational inference in models where sparsity and non-negativity are appropriate. We demonstrate the method on a recently proposed gamma process model for network data, as well as a novel sparse factor analysis. We outperform generic sampling algorithms and the approach of using Gaussian variational distributions on transformed variables.
1 Introduction
Bayesian models can be interpretable, but inference remains difficult to implement and scale to large datasets. This paper extends log-posterior-gradient variational inference to gamma latent variables and tests it in network and factor-analysis models.
1 Introduction
- Inference remains difficult to implement and scale, limiting practitioners’ ability to tailor Bayesian models and the use of existing packages on large real-world datasets.The cited packages include Infer.NET, WinBUGS, Church, and STAN.Source: §1 · p. 1
- Stochastic variational inference uses noisy, unbiased gradient estimates, which can be cheaper through data subsampling and enable faster convergence on larger datasets.It replaces coordinate-ascent variational Bayes updates with stochastic gradient descent.Source: §1 · pp. 1–2
- Gamma variational distributions extend stochastic gradient variational Bayes beyond Gaussian latent variables, addressing an open question while supporting sparsity and non-negativity.The paper presents a gamma process model for network data and a novel gamma process factor analysis model for continuous data.Source: §1 · pp. 2–3
- Prior log-posterior-gradient methods demonstrated effectiveness for multivariate Gaussians, while gamma variational distributions are a natural fit for sparse, non-negative latent variables.The paper also notes that gamma random variables underlie common Bayesian nonparametric priors such as the Dirichlet process.Source: §1 · p. 2
- The introduction contrasts the approach with recognition-model variational autoencoders and transformed Gaussian variational posteriors, noting limitations and a preference for gamma posteriors when priors are gamma.The stated VAE limitations include sensitivity to recognition-model choice, demonstration only for Gaussian latent variables, difficulty handling missing observations, and MLE-only model-parameter fitting.Source: §1 · p. 2
2 Methods
The methods optimize a constrained variational posterior by Monte Carlo gradient estimation, using a reparameterization suited to gamma variables and adaptive optimization for fitting.
2.1 Variational inference
- The variational posterior is constrained to a parametric form, specifically a product of gamma distributions, so maximizing the ELBO provides a tractable target for minimizing KL divergence.The general minimizing distribution is assumed intractable; the paper uses simplifying constraints on q.Source: §2.1 · p. 3
2.2 SGVB for continuous latent variables
- Reparameterized Monte Carlo estimates use the gradient of the objective with respect to the sample, and the authors report that one sample is generally sufficient.The reparameterization expresses x as ψ(z, θ) for a random variable z whose distribution does not depend on θ.Source: §2.2 · p. 4
2.3 Gamma variational distributions
- For gamma variables, the method uses a generic CDF transform because no simple transformation is available, and the shape-parameter gradient lacks an analytic form in commonly available special functions.The paper uses finite differences for moderate shape values and describes alternative approximations for small and large values.Source: §2.2 · p. 4, §2.3 · pp. 4–5
2.4 Optimization
- The gamma shape and rate are mapped through a softplus reparameterization to keep them positive, while momentum and adaptive learning-rate methods are considered for optimization.The tested optimizers include AdaGrad, RMSprop, and AdaDelta.Source: §2.4 · p. 5
3 Models
Section 3 assesses gamma stochastic gradient variational Bayes (SGVB) in network and factor-analysis models with gamma latent variables; the authors note that mixed Gaussian and gamma latent-variable models are also straightforward to implement. The examples use gamma-process constructions in the network model and factor analysis.
3 Models
- Algorithm 1 updates variational parameters using gradients of the log joint density minus the log variational density, with a step size such as AdaDelta.The algorithm transforms parameters before and after updates and repeats until convergence.Source: §3 · p. 6
3.1 Infinite edge partition model
- The infinite edge partition model represents edge probabilities through latent Poisson counts with rates r_k w_ik w_jk, linking nodes when the summed count is positive.Its weights W are positive, and r is a K-vector of real-valued parameters.Source: §3.1 · p. 6
- The model’s link function supports sparse graphs and evaluates the likelihood in time linear in observed present and missing edges, typically far below N^2.The finite-K distribution on r approximates a gamma process.Source: §3.1 · pp. 6–7
3.2 Gamma process factor analysis
- Gamma priors on factor loadings W enforce positivity, while integrating out Gaussian latent scores permits arbitrary real-valued observations in this semi-nonnegative factor analysis.The passage connects semi-nonnegative matrix factorization to k-means clustering and contrasts it with classical NMF, which requires positive data.Source: §3.2 · p. 7
- After precomputing YY^T, factor-analysis iterations cost O(D^3) independently of N, and W receives a hierarchical gamma-process prior.Source: §3.2 · p. 7
4 Results
Across network and factor-analysis experiments, gammaSGVB performs competitively with specialized methods and compares favorably with simpler or more generic alternatives in several settings. Results also show that optimizer choices and data size affect the comparisons.
4.1 Infinite edge partition model
- Adadelta with momentum achieves the best ELBO after 1,000 iterations, and performance is similar across tested ρ and momentum ranges, unlike the degraded no-momentum setting.RMSprop performs poorly; the similar-ELBO range is 1 − λ from 0.3 to 0.03 and ρ from 0.684 to 0.99.Source: §4.1 · p. 8, Figure 1 · p. 8, Figure 2 · p. 9
- GammaSGVB improves over the infinite relational model, while tuned MCMC performs best; MAP and NormSGVB perform poorly, especially at larger K.The NIPS evaluation uses 10 training-test splits and varies truncation level K.Source: §4.1 · p. 9, Figure 3 · p. 9
4.2 Gamma process factor analysis
- For small sample sizes N ≤ 200, NSFA typically slightly outperforms GPFA; as N increases, NSFA performance degrades at 1000 MCMC iterations, while gammaSGVB inference becomes easier.The true factor-loading elements are nonzero with probability 0.2.Source: §4.2 · p. 10
- GPFA outperforms SPCA across the synthetic-data settings, and by N = 10^4 SPCA is slower despite being faster at small N.GPFA runtime is stated to have no dependence on N, whereas SPCA runtime is approximately linear in N.Source: §4.2 · p. 10, Figure 4 · p. 10
- For fewer than around 2,000 CyTOF training points, GPFA outperforms both empirical covariance and Ledoit-Wolf shrinkage on test-set perplexity.The figure plots perplexity as training sample size increases.Source: Figure 5 · p. 10, §4.2 · p. 11
- On CyTOF data, GPFA models covariance structure while regularizing off-diagonal components; its nonnegative top loadings are easier to interpret than PCA loadings.Figure 6 presents empirical covariance, GPFA-estimated covariance, and the top 10 latent factor loadings.Source: Figure 6 · p. 11, §4.2 · p. 11
5 Discussion
The discussion argues that stochastic variational methods broaden scalable Bayesian inference: minibatches improve scalability, while Monte Carlo estimation handles a wider range of models than standard VBEM. The paper applies these ideas to sparse continuous gamma latent variables and reports an inference advantage from collapsing models.
5 Discussion
- The paper shows that SGVB applies to sparse continuous latent variables represented with a gamma variational posterior.It also reports that handling non-conjugate likelihoods can make collapsing models improve inference performance.Source: §5 · p. 12
- Minibatch-based SVI improves scalability, while Monte Carlo estimation lets SGVB handle a wider range of models than standard VBEM.Source: §5 · p. 12
- The only model-specific derivation required is the gradient of the log joint, which the discussion says is no more than LBFGS or HMC requires.The authors suggest automatic differentiation tools such as Theano for obtaining these gradients.Source: §5 · p. 12
- The authors identify combining these ideas with a variational autoencoder as a potential direction for scalable nonlinear sparse latent-variable models with some ability to model posterior dependencies.Source: §5 · p. 12