PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-box $α$-divergence Minimization

José Miguel Hernández-Lobato, Yingzhen Li, Mark Rowland, Daniel Hernández-Lobato, Thang Bui, Richard E. Turner

arXiv:1511.03243v3stat.ML

TL;DR

BB-α addresses memory and convergence obstacles in EP while extending α-divergence inference to complicated probabilistic models. It uses an optimizable energy function, stochastic gradient descent, Monte Carlo estimation, and automatic differentiation, with reported uncertainty estimates better than VB.

  • Problem

    EP can require prohibitive memory for large datasets and lacks convergence guarantees for efficient message-passing implementations.

  • Method

    BB-α constrains power EP site approximations and directly optimizes its energy function using stochastic optimization, Monte Carlo estimation, and automatic differentiation.

  • Results

    BB-α can provide approximate posteriors with better uncertainty estimates than VB and was evaluated on probit regression and Bayesian neural-network regression and classification.

  • Takeaways & Limitations

    BB-α supports memory-efficient, scalable approximate inference and can be implemented for new probabilistic models by supplying likelihood factors and their gradients.

  • Takeaways & Limitations

    The Monte Carlo stochastic gradients are biased by the logarithm applied to estimated integrals, although the bias becomes almost negligible relative to minibatch variance when K ≥10 in the considered models.

Abstract

from arXiv · show

Black-box alpha (BB-$α$) is a new approximate inference method based on the minimization of $α$-divergences. BB-$α$ scales to large datasets because it can be implemented using stochastic gradient descent. BB-$α$ can be applied to complex probabilistic models with little effort since it only requires as input the likelihood function and its gradients. These gradients can be easily obtained using automatic differentiation. By changing the divergence parameter $α$, the method is able to interpolate between variational Bayes (VB) ($α\rightarrow 0$) and an algorithm similar to expectation propagation (EP) ($α= 1$). Experiments on probit regression and neural network regression and classification problems show that BB-$α$ with non-standard settings of $α$, such as $α= 0.5$, usually produces better predictions than with $α\rightarrow 0$ (VB) or $α= 1$ (EP).

1. Introduction

Approximate inference is needed because Bayesian posteriors are often intractable, while existing methods face trade-offs in memory, convergence, and scalability. BB-α addresses these issues with factor tying, an analytic energy objective, stochastic optimization, and Monte Carlo black-box gradients.

  • 1. Introduction: Bayesian inference averages predictions under a posterior, but computing that posterior is often intractable for useful probabilistic models.VB and EP instead optimize tractable approximations through energy functions.
  • 1. Introduction: EP can outperform VB in some settings, but it requires storing local likelihood approximations and may lack convergence guarantees.Convergent double-loop alternatives exist, but they can be much slower than message passing.
  • 1. Introduction: SEP/AEP reduce EP's memory burden through factor tying, but their unknown energy function makes convergence analysis difficult.BB-α retains factor tying while defining an explicit energy objective.
  • 1. Introduction: BB-α applies factor tying directly to the power EP energy, yielding memory efficiency and direct optimization with stochastic gradient descent.Its analytic energy avoids double-loop procedures and supports large-scale learning with minibatches.
  • 1. Introduction: BB-α is a black-box method for complicated probabilistic models whose VB, EP, and power EP energies lack analytic forms.It uses Monte Carlo energy approximations while retaining factor tying, unlike related approaches based on costly double loops or non-convergent message passing.

2. α-Divergence and power EP

α-divergence minimization provides a family of approximate-inference objectives whose behavior changes with α, connecting VB-like and EP-like solutions. Power EP minimizes local divergences through moment matching, while its convergent double-loop implementation can be impractically slow.

  • 2. α-Divergence and power EP: The α-divergence family includes KL divergence and has a symmetric member at α = 1/2.The paper uses α-divergence to measure similarity between distributions.
  • 2. α-Divergence and power EP: Large positive α encourages a Gaussian approximation to cover all modes, whereas α →−∞ attracts it toward the mode with the largest probability mass.These behaviors describe global approximation by minimizing Dα[p||q].
  • 2. α-Divergence and power EP: For α ≥1, global α-divergence minimization may spread mass across low-probability regions when the true posterior is multimodal.The paper therefore emphasizes power EP's local α-divergence minimization instead.
  • 2. α-Divergence and power EP: Power EP iterates through cavity construction, tilted-distribution formation, moment matching, and site recovery to update the approximation.A convergent double-loop alternative exists for normal EP, but alternating optimization often requires too many iterations in practice.
  • 2. α-Divergence and power EP: At convergence, power EP's approximate posterior minimizes α-divergences from all tilted distributions to the approximation.Its moment-matching step is equivalent to minimizing an α-divergence with each tilted distribution as target.

3. Approximate local minimization of α-divergences

BB-α applies factor tying directly to a simplified power EP energy, yielding a memory-efficient objective that supports gradient-based and stochastic optimization. It uses Monte Carlo estimation and automatic differentiation to extend local α-divergence minimization to complex models.

  • 3. Approximate local minimization of α-divergences: BB-α applies factor tying directly to the power EP energy, producing an analytic objective that avoids double-loop procedures and supports gradient descent.The method ties local site parameters and rewrites the energy using the global approximate-posterior parameter.
  • 3. Approximate local minimization of α-divergences: The tied formulation averages moments from tilted distributions, unlike SEP’s averaging of natural parameters, and can mode-average depending on the application.The correspondence between natural and mean parameters relies on a minimal exponential-family assumption.
  • 3. Approximate local minimization of α-divergences: BB-α inherits factor tying’s memory efficiency while retaining an explicit energy that enables convergence analysis, stochastic optimization, adaptive learning rates, and hyper-parameter optimization.The method maintains only global and prior parameters rather than separate local approximations.
  • 3. Approximate local minimization of α-divergences: The energy is bounded below, and under the stated conditions stochastic optimization converges to a fixed point of the exact energy function.The lower-bound result holds for α ≤ N when the energy is finite; mini-batch gradients are used for large datasets.
  • 3. Approximate local minimization of α-divergences: BB-α approaches variational Bayes as α approaches zero and continuously interpolates toward an EP-like approximation as α varies between zero and one.The α-divergence is smooth on [0, 1], supporting this interpolation.
  • 3.2. Black-box α-divergence minimization: Monte Carlo samples estimate otherwise intractable expectations, while reparameterization reduces estimator variance and automatic differentiation supplies gradients for implementation.The resulting stochastic gradients are biased by the logarithm applied to the Monte Carlo estimator; experiments report nearly negligible bias for K ≥ 10 in the tested models.

4. Experiments

Experiments compare BB-α across probit regression, neural-network regression and classification, and a large molecular dataset, finding that the best α depends on the task while α=0.5 often performs well.

  • Probit regression: In probit regression, all methods perform similarly overall, with BB-α at α = 1.0 slightly better and α = 10−6 matching BB-VB.The comparison averages test log-likelihood, test error, and method rank across 50 train/test splits.
  • Neural network regression: Across neural-network regression, no tested α setting dominates every problem, but α = 0.5 produces good overall results.BB-α = 10−6 is almost indistinguishable from BB-VB, while regression-error differences are not significant in the reported UCI experiments.
  • Neural network classification: α = −1 achieves both higher test log-likelihood and lower test error than all tested non-negative α settings on MNIST.The comparison uses average test error and test log-likelihood over 20 random initializations.
  • Clean energy project data: BB-α with α = 0.5 is the best method on the Clean Energy Project dataset for both test error and test log-likelihood.Results are summarized over 20 random initializations.
  • Gradient bias and variance: Gradient bias decreases with more Monte Carlo samples and smaller α, and at K = 10 it is negligible relative to minibatch-induced gradient variability.The variability measure is reported as up to 5 orders of magnitude larger than the K = 10 bias.

5. Conclusions and future work

The paper concludes that BB-α approximates power EP through an optimizable energy function, supporting scalable stochastic optimization and application to complex likelihoods, while leaving α selection for future work.

  • 5. Conclusions and future work: BB-α approximates power EP by constraining site approximations and locally minimizing an α-divergence through an energy function.The α-divergence family includes the Kullback-Leibler divergence.
  • 5. Conclusions and future work: Stochastic gradient descent with minibatches provides scalability to large datasets without inefficient double-loop algorithms.The proposed energy has an analytic form that can be optimized directly.
  • 5. Conclusions and future work: Monte Carlo approximation and automatic differentiation allow BB-α to handle probabilistic models with complex likelihood factors.The method is presented as applicable across a wide range of probabilistic models.
  • 5. Conclusions and future work: The experiments demonstrate accuracy and scalability, but the optimal α setting may vary across tasks.The authors identify theoretical guidance or automatic α-selection tools as future work.

B. Linear regression example

The linear-regression example uses an analytically tractable model to compare BB-α with the true posterior, including a mean-field Gaussian approximation with diagonal covariance.

  • B. Linear regression example: The toy study compares the BB-α optimum with the true posterior when the posterior is representable and under a mean-field Gaussian family.The example is designed to provide intuition for BB-α in more general settings.
  • B. Linear regression example: The model is two-dimensional linear regression with Gaussian noise and a zero-mean Gaussian prior on the weights.Its simplicity permits analytical optimization of the BB-α energy function.
  • B. Linear regression example: Under the specified model, the posterior over weights is Gaussian with analytically determined mean and covariance.The posterior form follows from the linear regression model and prior.

B.1. Non-recovery of posterior distribution

For a Gaussian variational family containing the true posterior, BB-α’s optimality equations use tilted distributions, but moment matching generally fails to recover the posterior for non-zero α.

  • B.1. Non-recovery of posterior distribution: The variational family is defined by a Gaussian factor f(θ) combined with the prior, parameterized by natural parameters Λ and η.
  • B.1. Non-recovery of posterior distribution: Each data point produces a tilted distribution ep_n(θ) proportional to q(θ)(p(y_n|θ,x_n)/f(θ))^α for the optimality equations.
  • B.1. Non-recovery of posterior distribution: For the two-dimensional Gaussian family, the sufficient statistics determine the mean and covariance quantities used in the moment-matching equations.
  • B.1. Non-recovery of posterior distribution: Power EP returns the true posterior when it lies in the approximating family, but BB-α moment matching generally does not for non-zero α.The mismatch follows because mapping moments to natural parameters is nonlinear.

B.2. Example 1

A two-observation toy dataset makes BB-α’s non-recovery behavior explicit even when the variational family contains the true posterior and has diagonal covariance.

  • B.2. Example 1: The toy dataset uses x_1=(1,0)^⊤ and x_2=(0,1)^⊤, with arbitrary outputs and a true posterior having diagonal covariance.
  • B.2. Example 1: BB-α fits a variational family containing the true posterior to demonstrate that non-zero α generally does not recover it.
  • B.2. Example 1: The diagonal-precision restriction reduces the sufficient statistics while retaining the same BB-α optimality equation.
  • B.2. Example 1: With N=2 and the selected inputs, the first optimality equation yields linear equations in η, allowing η_i to be expressed in terms of λ_i.

B.2.2. MATCHING SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION

For zero outputs, the second-moment equations admit a constrained analytic solution whose fitted variance increases continuously and monotonically with α over (0,2), while the α→0 fit recovers the true posterior.

  • B.2.2. MATCHING SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION: The second-moment calculation uses only diagonal elements because the variational Gaussian has diagonal covariance.
  • B.2.2. MATCHING SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION: For zero outputs, the fixed-point equations simplify enough to yield an analytic solution under validity constraints on q and the tilted distributions.
  • B.2.2. MATCHING SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION: The fitted distribution’s variance increases continuously and monotonically with α over the range (0,2).This behavior is shown by plotting a diagonal covariance element against α.
  • B.2.2. MATCHING SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION: As α→0, BB-α recovers the true posterior, whereas fitted covariance parameters differ from the true posterior for non-zero α.The α=10^-6 fit is indistinguishable from the true posterior at the plotted scale.

B.3. Mean-field approximations

The mean-field analysis studies BB-α when the true posterior has non-diagonal covariance but the fitted Gaussian is constrained to have diagonal covariance.

  • B.3. Mean-field approximations: The example sets x_1=(1,−1)^⊤ and x_2=(−1,1)^⊤, producing a true posterior with non-diagonal covariance.
  • B.3. Mean-field approximations: The fitted variational family remains constrained to diagonal precision, and therefore diagonal covariance.
  • B.3. Mean-field approximations: Substituting the two observations into the first optimality equation yields a linear system for the natural parameters.
  • B.3. Mean-field approximations: When y_1=y_2, solving that system gives η=0, despite the generally complicated rational dependence on λ_1 and λ_2.

B.4.2. MATCHING THE SECOND MOMENTS OF THE VARIATIONAL DISTRIBUTION

For the symmetric zero-output case, the fitted covariance depends continuously and monotonically on α, while increasing α broadens the approximate posterior toward the true posterior.

  • Under symmetric outputs, λ1 and λ2 are equal, reducing the covariance derivation to a single precision parameter λ.
  • For y1 = y2 = 0, the resulting equation for λ is quadratic and therefore has an analytically expressible solution subject to positivity constraints.
  • The fitted covariance’s diagonal element changes continuously and monotonically with α over (0, 2).
  • As α approaches 0, BB-α mimics the KL fit with lower variance than the true posterior, while larger α increases distributional spread.

C. Results on toy dataset with neural network regression

The neural-network toy experiment evaluates BB-α predictive distributions across α using noisy cubic data, with the resulting predictions displayed for comparison.

  • The experiment uses 20 uniformly sampled inputs from [−4, 4] and targets y = x^3 + ϵ_n with ϵ_n ∼ N(0, 9).
  • A one-hidden-layer neural network with 100 rectifier units is fitted with output-noise variance σ^2 = 9 and weight-prior variance one.
  • Smaller and negative α values produce lower predictive variance in the reported toy neural-network experiment.
  • Figure 9 compares predictive distributions produced by neural networks trained with different α values on the toy dataset.
Loading 1511.03243v3…