Bayesian Learning for Neural Networks: an algorithmic survey

Martin Magris; Alexandros Iosifidis

Source-linked AI summary

Bayesian Learning for Neural Networks: an algorithmic survey

Martin Magris, Alexandros Iosifidis

arXiv:2211.11865v4Nov 21, 2022stat.MLcs.LG

TL;DR

Bayesian neural-network learning is difficult to use because posterior inference is challenging in high-dimensional models and practical implementations involve many technical choices. The survey introduces Bayesian principles and organizes standard and recent inference algorithms, emphasizing variational inference, natural gradients, and implementation details. It concludes with an accessible algorithmic overview while identifying scope and approximation limitations of particular methods.

Problem
Posterior inference in high-dimensional Bayesian neural networks is computationally challenging, limiting the widespread practical use of Bayesian learning.
Method
The survey presents Bayesian neural networks and standard and recent inference methods, emphasizing variational inference, natural gradients, manifold optimization, and practical algorithmic implementation.
Results
The survey provides an algorithmic overview of Bayesian neural-network methods with pseudocodes, update rules, and practical guidance for implementation.
Takeaways & Limitations
The survey supplies accessible tools for understanding and implementing a broad range of Bayesian neural-network inference methods.
Takeaways & Limitations
Bayesian neural-network inference remains difficult in practice, and particular methods impose restrictions such as Gaussian variational posteriors with diagonal covariance.

Abstract

from arXiv · show

The last decade witnessed a growing interest in Bayesian learning. Yet, the technicality of the topic and the multitude of ingredients involved therein, besides the complexity of turning theory into practical implementations, limit the use of the Bayesian learning paradigm, preventing its widespread adoption across different fields and applications. This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective. Upon providing a general introduction to Bayesian Neural Networks, we discuss and present both standard and recent approaches for Bayesian inference, with an emphasis on solutions relying on Variational Inference and the use of Natural gradients. We also discuss the use of manifold optimization as a state-of-the-art approach to Bayesian learning. We examine the characteristic properties of all the discussed methods, and provide pseudo-codes for their implementation, paying attention to practical aspects, such as the computation of the gradients.

1 Introduction

Bayesian learning offers uncertainty-aware alternatives to point estimation, but posterior computation remains difficult in high-dimensional neural networks. This survey presents Bayesian neural networks and practical inference methods, emphasizing variational inference and algorithmic implementation.

1.1 The Bayesian paradigm: Bayesian methods represent uncertainty through posterior distributions, can reduce overfitting, and support sequential learning while retaining prior knowledge.The posterior combines prior beliefs with evidence from observed data.
1 Introduction: Posterior computation is the central challenge because traditional sampling becomes prohibitive with many parameters and large datasets.Variational Inference and natural gradients are presented as scalable alternatives for Bayesian learning.
1 Introduction: The survey organizes standard and recent Bayesian neural-network methods into an accessible algorithmic guide with implementation-oriented update rules and pseudocodes.It focuses on Bayesian methods and excludes architectures that resemble Bayesian models without being implicitly Bayesian.
1.3 Motivation for adopting Bayesian Neural Networks: Bayesian neural networks learn distributions over weights rather than fixed values, enabling epistemic uncertainty to decrease as additional data address missing knowledge.This framework also combines expert knowledge encoded in priors with experimental evidence.
1.4 Bayesian Neural Networks: A BNN requires a neural architecture and likelihood linking inputs, parameters, and outputs before posterior inference can be performed.Gaussian likelihoods are given for regression and Binomial likelihoods for classification.
1.5 Variational Inference (VI): Variational Inference approximates the posterior with a tractable distribution q(θ) and minimizes its KL divergence from the true posterior.This KL minimization is equivalent to maximizing a lower bound on the log evidence, with a chosen parametric variational family.

2 Sampling methods

Sampling methods approximate Bayesian posteriors by constructing Markov chains, with Metropolis-Hastings and Hamiltonian Monte Carlo offering different strategies for exploring high-dimensional parameter spaces.

2.1 Monte Carlo Markov Chain (MCMC): MCMC obtains posterior samples by constructing a Markov chain whose stationary distribution is the desired target, but early samples may be unrepresentative.Successive chain samples can also remain autocorrelated, reducing the effective sample size and inflating limiting variance.
2.2 Metropolis-Hastings (MH): Metropolis-Hastings samples a target distribution using proposals accepted according to the ratio of proposed and current densities.The proposal density is often symmetric and Gaussian-centered on the current state; higher-density proposals are always accepted, while others are accepted probabilistically.
2.2 Metropolis-Hastings (MH): MH remains applicable in high dimensions because it does not suffer from the increasing rejection rate associated with the curse of dimensionality.This property makes MH suitable for large Bayesian inference problems such as training Bayesian neural networks.
2.3 Hamiltonian Monte Carlo (HMC): Hamiltonian Monte Carlo augments the target density with an auxiliary momentum variable and uses approximate Hamiltonian dynamics to generate efficient transitions.The method typically uses Gaussian momentum, Hamilton’s equations, automatic differentiation, and a leapfrog integrator before an MH-like correction.
2.3 Hamiltonian Monte Carlo (HMC): HMC repeats leapfrog updates for L steps of size ε, then accepts or rejects the resulting state through a Metropolis-Hastings criterion.The hyperparameters L and ε determine the simulated trajectory and are often difficult to tune; HMC also suffers from multimodality.

3 Monte Carlo Dropout (MCD)

Monte Carlo Dropout interprets stochastic neuron masking as variational inference in a deep Gaussian process, enabling predictive uncertainty estimates through repeated stochastic forward passes. Its posterior restriction and training behavior impose important limitations.

3 Monte Carlo Dropout (MCD): Dropout-based training is equivalent to minimizing the KL divergence between an approximate variational model and a deep Gaussian process posterior.The approximation uses Bernoulli-masked weight matrices and can be optimized with a feasible unbiased Monte Carlo estimator.
3 Monte Carlo Dropout (MCD): Predictive means, variances, and higher-order statistics are estimated from repeated forward passes with independently sampled dropout masks.The predictive mean uses Ns stochastic network evaluations, while moment matching estimates additional predictive statistics.
3 Monte Carlo Dropout (MCD): MCD constrains variational inference to a specific posterior form associated with a deep Gaussian process, which may not fully capture model uncertainty.The survey also notes improper-prior and singular-posterior issues in this approximation.
3 Monte Carlo Dropout (MCD): High dropout rates can slow convergence, lengthen training, and cause training examples to receive little relative importance.These costs arise despite dropout requiring no additional effort compared with traditional neural-network training.

4 Bayes-By-Backprop (BBB)

Bayes-By-Backprop performs variational Bayesian inference by reparameterizing weight samples and optimizing the resulting stochastic objective with ordinary backpropagation. Its compatibility with standard optimizers makes Bayesian learning practical for neural networks.

4 Bayes-By-Backprop (BBB): The sampled variational objective log q(w|θ) − log p(w)p(D|w) provides a stochastic approximation to the KL objective minimized by variational inference.The objective is evaluated after sampling ε and obtaining the transformed weights.
4 Bayes-By-Backprop (BBB): BBB samples parameter-independent noise ε and deterministically transforms it into weights w=t(θ,ε), enabling unbiased stochastic gradient estimates.Because ε is independent of the variational parameters, a single draw can support backpropagation-based gradient estimation.
4 Bayes-By-Backprop (BBB): Bayes-By-Backprop makes variational Bayesian inference feasible by combining stochastic weight sampling with standard backpropagation and optimization algorithms.The survey identifies this compatibility as BBB’s major breakthrough element.
4 Bayes-By-Backprop (BBB): For a diagonal Gaussian posterior, BBB uses mean µ and covariance σ^2I, with σ parameterized as log(1 + exp(ρ)) to remain non-negative.The variational parameters are θ=(µ,ρ), and the corresponding algorithm updates them through sampled gradients.

5 Exponential family and Natural gradients

Natural gradients replace Euclidean updates with geometry induced by the Fisher information metric, better matching optimization to the space of probability distributions. The survey presents their update scheme and explains why they address instability in ordinary gradient search.

5 Exponential family and Natural gradients: In a minimal exponential-family representation, the log-partition function is convex and the Fisher information matrix is positive definite and invertible.The resulting one-to-one mapping connects natural and mean parameters.
5 Exponential family and Natural gradients: Natural gradients pre-multiply the Euclidean gradient by the inverse Fisher information matrix, replacing Euclidean distance with a distribution-aware Riemannian metric.This geometry is induced by the exponential-family distribution represented by the variational parameters.
5 Exponential family and Natural gradients: Plain gradient search becomes unstable as σ decreases because update variance grows proportionally to 1/σ, whereas natural gradients address this issue.Extreme initial σ values respectively produce tiny or very large unstable updates.
5 Exponential family and Natural gradients: Natural-gradient updates follow ζ=ζ+βI^-1∇ζL, where I rescales the gradient according to local distribution geometry.The survey gives this as the generic stochastic-gradient scheme with natural gradients.

6 Black-Box methods

Black-box variational methods avoid model-specific gradient calculations by estimating variational objectives from samples, while NG-BBVI adds natural-gradient updates and variance-reduction machinery. This broadens applicability but can make computation more involved.

6 Black-Box methods: BBVI estimates ELBO gradients from variational-posterior samples using the log-derivative trick, without requiring model gradients.The resulting estimator depends on log q derivatives and evaluations of the joint density and variational density.
6 Black-Box methods: NG-BBVI combines black-box ELBO gradients with natural gradients, control variates, Rao-Blackwellization, and an ADAM-like update.Its implementation splits posterior samples between control-variate estimation and independent Monte Carlo gradient estimation.
6 Black-Box methods: NG-BBVI is favored over BBVI because the survey considers natural-gradient optimization preferable for the optimization reasons discussed earlier.Both methods use control variates and Rao-Blackwellization for variance reduction.
6 Black-Box methods: The black-box framework gives NG-BBVI general applicability across variational-posterior forms, but its Monte Carlo Fisher-information computations remain approximate.The Fisher information can also be a computational bottleneck when direct inversion is required.

7 Natural gradient methods for exponential-family variational distributions

Natural-gradient methods exploit exponential-family structure to simplify Bayesian variational updates, while successive methods reduce objective-gradient or model-curvature costs. These approaches also expose practical trade-offs involving covariance constraints and Hessian approximations.

Natural-gradient foundations: Exact natural gradients for exponential-family distributions can be computed through expectation-parameter Euclidean gradients, avoiding explicit Fisher information matrix computation and inversion.The natural/expectation-parameter duality provides the theoretical basis for this simplification.
Natural-Gradient Variational Inference: Natural-Gradient Variational Inference updates Gaussian posterior means and covariances without computing the Fisher information matrix, but full-covariance updates may fail to preserve positive definiteness.The mean update also uses the one-step-ahead covariance information and can require an online matrix inversion.
Variational Online Newton: Variational Online Newton removes variational-objective gradients from NGVI, requiring only model log-likelihood gradients and Hessians computable through standard backpropagation.This targets a central computational burden in NGVI while retaining its natural-gradient development.
Stochastic curvature estimation: VON-style curvature updates use stochastic gradients and Hessians from sampled observations or mini-batches, with larger mini-batches lowering estimator variance but the squared-gradient Hessian approximation introducing bias.The reported bias makes increasing mini-batch size inadvisable for that approximation.
Variational Online Gauss-Newton: VOGN is presented as feasible for large-scale Bayesian deep learning with practical support for normalization, augmentation, momentum, and distributed computation, but it is limited to diagonal-covariance Gaussian posteriors.Its stated feasibility on large datasets and deep architectures motivates its characterization as a state-of-the-art method in the survey.
Quasi Black-Box Variational Inference: QBVI combines black-box score estimation with exact exponential-family natural-gradient computation, avoiding sampling-based natural gradients while retaining model-specific gradient and Hessian requirements.The method converts natural-parameter gradients into Euclidean gradients of the lower bound.

8 Variational Inference on manifolds

Manifold optimization addresses constrained variational updates by operating on the valid parameter manifold rather than unrestricted Euclidean space. Its Riemannian workflow projects gradients into tangent spaces, retracts updates onto the manifold, and supports transport between tangent spaces.

Motivation: Manifold optimization is motivated by the need to keep variational covariance matrices symmetric and positive definite during optimization.This constraint is central for Gaussian and inverse-Wishart variational settings.
Riemannian optimization: Unlike Euclidean updates, Riemannian optimization seeks parameters on a manifold and uses tangent-space geometry to define valid descent directions under the manifold constraint.The approach treats optimization as constrained motion on a smooth surface such as the SPD set.
Tangent spaces and retractions: Tangent spaces provide local vector-space approximations to manifolds, enabling retractions that transform tangent-space steps into updates on the original manifold.This local structure supports optimization despite the manifold not generally being a vector space.
RSGD components: RSGD evaluates an ambient gradient, projects it onto the tangent space, and takes a retracted step; vector transport then relates tangent vectors across different manifold points.The three-step RSGD procedure corresponds to the projection, retraction, and transport elements illustrated in Figure 4.
Riemannian gradients: Riemannian gradients are obtained by projecting ambient gradients onto the manifold’s tangent space, while Gaussian manifolds make this projection the identity in the described representation.The general projection can be difficult, although it is trivial in the Gaussian-manifold case discussed.

9 Conclusion

The survey presents Bayesian learning for neural networks through an accessible algorithmic treatment spanning foundational concepts, practical inference methods, and manifold optimization. It aims to clarify methodological connections and support implementation and further adoption.

Survey scope: The survey provides an algorithmic overview of standard and recent Bayesian neural-network methods, including pseudocodes and update rules for implementation.Its presentation emphasizes core concepts, methodological differences, and practical algorithmic details.
Methodological coverage: The methods covered range from Monte Carlo and Monte Carlo Dropout to variational inference, natural-gradient optimizers, and manifold-based Gaussian variational methods.The survey specifically discusses BBB, BBVI, NG-BBVI, NGVI, VON, VOGN, QBVI, and manifold variational solutions.
Conclusion: The authors state that the comprehensive algorithmic treatment is intended to clarify connections and differences among Bayesian neural-network methods and promote their use and further research.This is presented as the survey’s intended contribution rather than as an empirical benchmark result.

A Nomenclature

This section presents a nomenclature for the mathematical symbols used in the paper.

Table 1 lists the most used mathematical symbols and their nomenclature.