PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Variational Inference in Stan

Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, David M. Blei

arXiv:1506.03431v2stat.ML

TL;DR

Variational inference is difficult to automate because its approximating families and calculations are model-specific. The paper introduces ADVI, which automates family selection and variational optimization for differentiable models in Stan. ADVI shows speed improvements across evaluated models and analyzes 250 000 images in under two hours, within its stated differentiability and support assumptions.

  • Problem

    Variational inference requires expert design of approximating families and model-specific calculations, making its algorithms difficult to derive and tied to each model and approximation.

  • Method

    ADVI transforms latent variables to unconstrained real space, uses a Gaussian variational distribution with automatic differentiation, and applies stochastic optimization in Stan.

  • Results

    ADVI provides speed improvements across evaluated models and produces an approximate posterior for 250 000 images in under two hours, while MCMC cannot practically begin analysis at that scale.

  • Takeaways & Limitations

    ADVI enables variational inference for models expressible in Stan and supports analysis of large datasets using stochastic minibatches.

  • Takeaways & Limitations

    ADVI focuses on differentiable probability models with continuous latent variables and valid gradients within the prior support, without requiring conjugacy.

Abstract

from arXiv · show

Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). The user only provides a Bayesian model and a dataset; nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We implement ADVI in Stan (code available now), a probabilistic programming framework. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

1 Introduction

Variational inference makes Bayesian approximation an optimization problem but is difficult to derive because its approximations and calculations are model-specific. ADVI automates this process in Stan and shows substantial speed and scalability advantages across models and datasets.

  • Variational inference approximates difficult posterior calculations with simpler densities, turning approximate Bayesian inference into optimization.
  • Deriving variational inference algorithms requires expert choices of the approximating family and model-specific calculations, tying each algorithm to both.
  • ADVI automatically selects a variational family and optimization algorithm for differentiable probability models expressed in Stan.
  • ADVI is orders of magnitude faster than NUTS on a Gaussian mixture model and consistently speeds up inference for factorization and hierarchical generalized linear models.
  • ADVI produces an approximate posterior in under two hours for 250 000 images, whereas MCMC techniques cannot practically begin analysis at that scale.

2 Automatic Differentiation Variational Inference

ADVI automates variational inference for differentiable probability models by transforming constrained latent variables, using a Gaussian approximation, and optimizing the ELBO with automatic differentiation and stochastic optimization.

  • Optimization: ADVI combines automatic differentiation with stochastic optimization because the ELBO expectation is otherwise an intractable integral depending on variational parameters.Elliptical standardization reparameterizes Gaussian scales using log standard deviations before optimization.
  • Supported models: ADVI targets differentiable probability models with continuous latent variables and requires no full or conditional conjugacy assumptions.The supported models provide gradients of the log joint within the prior support.
  • Variational objective: The variational objective maximizes an evidence lower bound whose terms are expected joint density and variational entropy, thereby minimizing KL divergence.Support matching is explicit because the approximating density must remain within the posterior support.
  • Automatic transformation: ADVI transforms latent-variable support to the real coordinate space, then uses a Gaussian variational density whose inverse transform yields a non-Gaussian original-space approximation.The transformation keeps the approximation within the posterior support.
  • Scalability: The stochastic extension reduces per-iteration complexity from O(2NMK) to O(2BMK) by using minibatches of size B ≪ N and scaling the likelihood by N/B.M is the number of Monte Carlo samples, typically between 1 and 10.

3 Empirical Study

The empirical study compares ADVI with HMC and NUTS across hierarchical regressions, matrix factorization, and Gaussian mixtures, including a 250,000-image dataset. ADVI generally matches predictive accuracy while offering substantial speed and scalability advantages.

  • Study design: The study compares ADVI with Hamiltonian Monte Carlo and NUTS using held-out predictive accuracy over time.The evaluation covers two hierarchical regression models, two matrix factorization models, and a mixture model.
  • Hierarchical regression: All hierarchical regression methods reach the same predictive accuracy, while one Monte Carlo sample per ADVI iteration is fastest.The models include automatic-relevance-determination linear regression and spatial hierarchical logistic regression.
  • Nonnegative matrix factorization: ADVI provides an order of magnitude speed improvement over NUTS on both nonnegative matrix factorization models.HMC produces no useful samples within one hour, and NUTS struggles with the Dirichlet Exponential model.
  • Nonnegative matrix factorization: The Gamma Poisson factors select significant frames, whereas Dirichlet Exponential factors identify sparse moving facial components such as eyebrows, cheeks, and the mouth.Both models use the 1956-frame Frey Faces dataset.
  • Gaussian mixture model: On 1,000 images, NUTS struggles to find an adequate mixture-model solution and HMC fails altogether, likely because of label switching.The model uses 10 mixture components in this smaller-data experiment.
  • Gaussian mixture model: With minibatches of 500 or larger, ADVI reaches high predictive accuracy on the full 250,000-image dataset and converges in about two hours.Smaller minibatches lead to suboptimal solutions.

4 Conclusion

ADVI combines automatic transformations, an implicit non-Gaussian variational approximation, and automatic differentiation in Stan. The authors present it as a tool for exploring many models and analyzing large datasets, and state that it is available for use.

  • ADVI in Stan combines automatic transformations, an implicit non-Gaussian variational approximation, and automatic differentiation.
  • The authors state that ADVI enables exploration of many models and analysis of large datasets with ease.
  • ADVI is available as part of Stan and is ready for anyone to use.

A Transformation of the Evidence Lower Bound

This section transforms the evidence lower bound from the original latent-variable space into real coordinates. It uses a Gaussian variational approximation, Jacobian adjustments, and gradients obtained through differentiation under the integral and the chain rule.

  • The variational approximation is transformed from the original latent-variable space into real coordinate space.
  • In real coordinates, the variational approximation is Gaussian, and its entropy yields the transformed evidence lower bound.
  • Gradients with respect to the standardization parameters use differentiation under the integral and the chain rule.The treatment considers both the µ and ω parameters, including a single-component ω_k gradient.
  • The multivariate change of variables uses the absolute determinant of the inverse Jacobian to account for volume changes.The Jacobian adjustment preserves proper density normalization under nonlinear transformations.

E Setting a Stepsize Sequence for advi

ADVI uses adaGrad to adapt its stepsize sequence, but limits adaGrad’s memory because random initialization can otherwise cause excessively small steps and slow convergence.

  • ADVI uses adaGrad to adaptively set the stepsize sequence during optimization.
  • Random initialization can place the variational approximation far from the true posterior, causing adaGrad to take very small steps thereafter.
  • Limiting adaGrad’s memory to recent iterations speeds convergence in practice.
  • The implementation computes the limited-memory statistic recursively to save memory.
  • Stan uses η = 0.1 and τ = 1 as default values.

F Linear Regression with Automatic Relevance Determination

The paper evaluates ADVI on automatic relevance determination linear regression, a high-dimensional sparse model with independently regularized weights and simulated sparse predictive structure.

  • Automatic relevance determination linear regression is a high-dimensional sparse regression model.
  • The model uses D-dimensional inputs, a one-dimensional output for each observation, and a D-dimensional weight vector.
  • Each weight receives its own independent Gamma prior through the D-dimensional hyper-prior α.
  • The simulations generate data in which only half of the regressions have predictive power.
  • The reported results use a0 = b0 = c0 = d0 = 1 as the Gamma-prior hyper-parameters.

G Hierarchical Logistic Regression

The paper studies hierarchical logistic regression as an intuitive model of dependencies, using voting preferences from the 1988 United States presidential election.

  • Hierarchical logistic regression models dependencies in an intuitive and powerful way.
  • The study applies this model to voting preferences from the 1988 United States presidential election.
  • The regression includes coefficients for female, black, and female-black interaction indicators.
  • The model uses the sigmoid function, also known as the logistic function.
  • Its hierarchical variables are indexed across k = 1, . . . , K, l = 1, . . . , L, and m = 1, . . . , M.
  • All variance terms have uniform hyper-priors constrained between 0 and 100.

H Non-negative Matrix Factorization: Constrained Gamma Poisson Model

The paper studies a constrained Gamma Poisson factorization model for discrete user-item matrices, addressing its non-unique parameterization by constraining each user-factor vector.

  • Gamma Poisson factorization analyzes discrete data matrices and is framed here as a user-item recommendation setting.
  • The observations form a U × I matrix, with users indexed by u and items indexed by i.
  • The generative process separately iterates over users, items, and user-item pairs.
  • Scaling θu by α and βi by α^-1 leaves the likelihood unchanged, so the model is not uniquely identifiable.
  • The experiments constrain each θu vector to be positive and ordered, set K = 10, and use Gamma hyper-parameters equal to 1.

I Non-negative Matrix Factorization: Dirichlet Exponential Model

The Dirichlet Exponential model is a non-conjugate factorization model for discrete data, designed to combine uniqueness with sparse, interpretable factors.

  • The Dirichlet Exponential model uses a Poisson likelihood and is non-conjugate.The paper notes that this model does not appear to have been studied in the literature.
  • The Dirichlet prior enforces uniqueness, while the exponential prior promotes sparsity.
  • The model is specified through a generative process over users, items, and their interactions.
  • With K = 10, α0 = 1000, and λ0 = 0.1, the resulting factors are sparse and appear interpretable.The Stan implementation is provided in Figure 9.

J Gaussian Mixture Model

The paper applies a high-dimensional Gaussian mixture model to natural-image color histograms, using Gaussian, lognormal, and Dirichlet priors and providing Stan implementations with optional subsampling.

  • The Gaussian mixture model groups natural images according to their color histograms.
  • The model uses Gaussian priors for mixture means, lognormal priors for standard deviations, and a Dirichlet prior for mixture components.
  • Each image is represented by a D-dimensional observation, with the imageclef color histograms having D = 576 dimensions.The 576 dimensions concatenate three 192-length histograms for the red, green, and blue channels.
  • The histograms are standardized to zero mean and unit variance, with α0 = 10 000 and σµ = 0.1 specified for the model.
  • Stan code is provided for the ADVI implementation and for a stochastic dataset-subsampling version.
Loading 1506.03431v2…