PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Neural Networks: An Introduction and Survey

Ethan Goan, Clinton Fookes

arXiv:2006.12024v3stat.MLcs.LG

TL;DR

Neural networks deliver state-of-the-art performance but commonly use frequentist point estimates that cannot reason about predictive uncertainty and may overfit. This paper introduces Bayesian Neural Networks, surveys seminal and approximate inference methods, and finds that Bayesian CNNs can match point-estimate CNN predictive performance while providing informative credible intervals, alongside persistent approximation limitations.

  • Problem

    Frequentist neural networks provide point estimates and can be overconfident, while their black-box decisions create concerns for responsible deployment.

  • Method

    The paper introduces BNNs, surveys seminal research and approximate inference methods, and uses literature review and experiments to examine their capabilities and limits.

  • Results

    Bayesian CNNs showed comparable predictive performance to point-estimate CNNs for image classification while providing informative credible intervals for difficult data points.

  • Takeaways & Limitations

    Credible intervals can provide an informative and intuitive measure of uncertainty for difficult-to-classify data points.

  • Takeaways & Limitations

    Fully factorised variational approaches can produce over-confident predictions because they sacrifice correlations between network parameters.

Abstract

from arXiv · show

Neural Networks (NNs) have provided state-of-the-art results for many challenging machine learning tasks such as detection, regression and classification across the domains of computer vision, speech recognition and natural language processing. Despite their success, they are often implemented in a frequentist scheme, meaning they are unable to reason about uncertainty in their predictions. This article introduces Bayesian Neural Networks (BNNs) and the seminal research regarding their implementation. Different approximate inference methods are compared, and used to highlight where future research can improve on current methods.

1 Introduction

Neural networks achieve strong results but frequentist point estimates can overfit, obscure decision processes, and provide over-confident predictions. Bayesian Neural Networks address these concerns by placing distributions over network parameters, while this survey introduces the field and reviews approximation methods and their limits.

  • Frequentist neural networks optimise a cost function to produce point estimates, but greater capacity can make them prone to overfitting.Large datasets and regularisation methods such as MAP estimation can limit complexity.
  • Neural networks have achieved state-of-the-art results in image classification, object detection, and speech recognition.
  • Neural networks are statistical black boxes whose uninterpretable decisions raise ethical concerns in social and safety-critical environments.The paper cites incidents involving gender and racial bias and, in extreme cases, loss of life.
  • Probabilistic methods provide output distributions in regions without training data, whereas neural networks provide point estimates.The distribution enables uncertainty identification and supports development of trustworthy models.
  • Bayesian Neural Networks place a distribution over network parameters to retain high model capacity while providing theoretical benefits of Bayesian analysis.
  • The article provides an accessible introduction, surveys seminal BNN research, and uses experiments to discuss current methods’ capabilities and limits.The survey is selective rather than exhaustive because the full literature could fill multiple textbooks.

2.1 Neural Networks

Neural networks transform inputs through weighted layers and nonlinear activations, with architecture and output activation adapted to regression or classification. Training uses backpropagation to update parameters from cost-function derivatives.

  • An MLP represents a network with input, hidden, and output layers, and can support binary classification or one-dimensional regression.Modern architectures such as convolutional networks have equivalent MLP representations.
  • Network weights encode the strengths of connections between neurons, and deeper layers take the preceding layer’s output as input.
  • Each hidden neuron applies an affine transformation followed by an element-wise activation function.Common activations include Sigmoid, TanH, ReLU, and Leaky-ReLU; the original sign activation was abandoned because its derivative is zero when defined.
  • Regression uses the identity output activation, whereas binary classification uses a Sigmoid output activation.
  • Backpropagation computes parameter derivatives and updates weights during training using a learning rate.The derivatives are obtained by applying the chain rule across network layers.
  • ReLU’s larger gradient can help prevent vanishing gradients in early layers of deep neural networks.

2.2 Bayesian Neural Networks

A Bayesian Neural Network treats unknown weights as latent random variables and infers their posterior distribution from prior beliefs and observed data. Predictions marginalise over this distribution, but exact inference is often intractable for nonlinear, high-dimensional networks.

  • Bayesian neural networks infer a posterior distribution p(ω|D) over hidden network weights conditional on observed training data.Bayes’ theorem expresses the distribution of weights in terms of observable data.
  • The prior describes beliefs about weights before observing data, while the likelihood is determined by the network architecture and loss function.For homoscedastic one-dimensional regression with mean squared error and known noise variance, the likelihood is Gaussian.
  • Predictions are expectations over the posterior, so predictive means, variances, and intervals are obtained by changing the quantity averaged.This averages a function weighted by the posterior distribution.
  • Marginalising over unknown weights yields predictions represented as valid conditional probabilities.This contrasts with the optimisation scheme used in the frequentist setting.
  • Additional unknown quantities, such as noise variance or prior parameters, can be treated as latent variables with priors and marginalised.
  • Exact posterior computation is often intractable because marginal likelihoods are analytically difficult for nonlinear or non-conjugate models and costly to integrate in high dimensions.Approximate posterior inference is therefore required.

2.3 Origin of Bayesian Neural Networks

Early BNN research framed Bayesian inference as a way to compare model hypotheses using evidence, while developing tractable approximations for neural-network posteriors. These approaches connected Bayesian model assessment with neural networks but faced restrictions from computational cost, posterior assumptions, covariance complexity, and shallow architectures.

  • Early Bayesian model design: The first BNN work interpreted neural-network loss functions statistically and showed that priors over weights enable posterior inference through Bayes’ theorem.
  • Early Bayesian model design: MacKay’s framework separated inference for fitting parameters from inference for assessing model suitability, using evidence to compare alternative model hypotheses.
  • Evidence and model comparison: The Occam factor compares posterior and prior plausible parameter-space widths, naturally penalizing unnecessarily complex models and less informative priors through reduced evidence.
  • Evidence and model comparison: Marginal-likelihood computation is expensive, making evidence-based comparison of many architectures potentially infeasible, although local minima can be assessed as alternative inference solutions.
  • Approximation limits: Analytic posterior approximations can reduce inference time, but early methods assumed factorised weight posteriors, incurred covariance-related parameter growth, or restricted networks to one hidden layer and specific activations.
  • Approximation limits: The surveyed methods helped merge backpropagation with Bayesian modelling, enabling complex regression tasks over large datasets to be handled probabilistically.

2.4 Modern BNNs

Modern BNN research focuses on scalable approximate inference for deep networks, balancing predictive accuracy, uncertainty estimation, and computational cost. Variational methods, gradient estimators, and alternative posterior updates address these challenges with varying limitations.

  • Modern variational inference: Modern BNN research shifted toward variational inference because it can use backpropagation-like optimization for complex, multilayer networks.Earlier analytic approximations targeted single-hidden-layer regression networks and became unsuitable for modern architectures.
  • Uncertainty estimation: Dropout gained theoretical justification as approximate Bayesian inference, strengthening the connection between uncertainty estimation and established neural-network regularization.The survey contrasts this later theoretical grounding with the initial Dropout paper’s reliance on empirical results and an unrelated motivating theory.
  • Gradient estimation: Large Monte Carlo gradient variance limits practical variational inference for complex BNNs and motivates research into lower-variance gradient estimates.Score-function estimators have considerable variance, whereas pathwise estimators based on reparameterization are favored for reduced variance.
  • Alternative approximate inference: Probabilistic Back Propagation updates parameter posteriors using forward-pass log-marginal probabilities and moment matching rather than traditional prediction-error objectives.The method maintains equivalent means and variances between approximate and variational distributions.
  • Alternative approximate inference: PBP achieved reasonable accuracy and uncertainty estimation on multiple small datasets compared with HMC for simple regression, but online training becomes prohibitive on large datasets.Its online formulation may suit some applications or sequential updates as additional data arrive.
  • Scalable training: Bayes by Backprop applies the reparameterization trick to obtain unbiased derivatives of expectations, while stochastic mini-batches make large-data variational training efficient.Sampling in the reparameterized space yields gradient-estimator variance that scales linearly with the number of mini-batches.

2.5 Gaussian Process Properties of BNNs

A Gaussian prior over parameters makes an infinitely wide single-hidden-layer network converge to a Gaussian-process prior over outputs. This connection motivates architectures and analyses that combine neural-network flexibility with Gaussian-process uncertainty and interpretability.

  • Infinite-width limit: As hidden-unit count approaches infinity, a single-hidden-layer network with Gaussian parameter priors induces a Gaussian-process prior over its output.The result assumes a regression model without a nonlinear activation on the output units.
  • Infinite-width limit: The central-limit argument explains the limit: the network output is a sum of random basis functions that becomes Gaussian as the number of hidden units grows.An infinite sum of basis functions gives the resulting output the interpretation of a Gaussian process.
  • Consequences of the connection: The BNN–GP link motivates combining neural networks’ predictive flexibility with Gaussian processes’ uncertainty, interpretability, and robustness.The survey notes that Gaussian processes incur reduced predictive performance and exponentially growing computational requirements as datasets grow.
  • Infinite-width limit: Figure 6 samples network outputs at f(0.2) and f(−0.4) while increasing hidden units from 1 to 100 to illustrate the induced GP prior.Panels correspond to 1, 3, 10, and 100 hidden units.
  • Deep extensions: Deep Gaussian processes stack individual GPs to learn non-Gaussian densities, but their Gram matrices and intractable marginal likelihoods create substantial computational challenges.Variational inference can make the approximation tractable and reduce complexity toward that of sparse GPs.
  • Deep extensions: A deep GP can be approximated with a BNN because concatenating multiple infinitely wide layers converges to a deep Gaussian process.Related work also studies how nonlinear activations determine GP covariance properties and how convolutional structure induces GP behavior.

2.6 Limitations in Current BNNs

Current BNN inference relies heavily on variational inference, whose simplifying assumptions reduce computational cost but can impair posterior accuracy. Alternative divergences and richer approximations may improve uncertainty estimates, yet practical use remains constrained by optimization and likelihood-computation requirements.

  • Variational inference limitations: Mean-field variational inference factorizes the approximate posterior by assuming parameter independence, reducing complexity at the cost of probabilistic accuracy.This assumption is central to MFVB and can omit posterior dependencies.
  • Variational inference limitations: VI commonly produces overconfident models whose predictive means are accurate while predictive variances are substantially underestimated.The survey reports this pattern across much current BNN research.
  • Alternative approximations: More expressive approximations, including mixtures and normalising flows, are proposed to capture complex posteriors, but invertibility constraints can limit flexibility.Normalising flows use sequences of invertible functions, whose numerical-stability restrictions can constrain posterior complexity.
  • Alternative divergences: The forward KL used in VI typically underestimates variance, whereas reverse KL often overestimates it; the Hellinger distance can provide a compromise.The squared Hellinger distance is symmetric and satisfies the triangle inequality, but cannot be directly used in VI because the marginal likelihood remains inseparable.
  • Computational trade-offs: VI remains dominant because it supports stochastic-gradient optimization and likelihood subsampling, whereas MCMC retains stronger theoretical guarantees but traditionally requires full-data updates.Stochastic-gradient MCMC has been proposed to reduce this computational burden.

3 Comparison of Modern BNNs

The survey compares Bayes by Backprop and MC Dropout as practical VI methods, examining their regression behavior against Gaussian processes and extending Bayesian modeling to convolutional networks. Their uncertainty estimates exhibit different bias–variance trade-offs, while BCNNs retain competitive classification performance and expose predictive uncertainty.

  • 3 Comparison of Modern BNNs: Bayes by Backprop and MC Dropout are highlighted as prominent BNN inference methods because both support stochastic-gradient training on practical data sets.Their prominence motivates direct empirical comparison.
  • 3 Comparison of Modern BNNs: The regression experiments use homoskedastic Gaussian likelihoods, spike-slab priors, method-specific approximate posteriors, and Monte Carlo integration for predictive quantities.Bayes by Backprop uses a fully factorised Gaussian posterior, while MC Dropout uses a scaled Bernoulli distribution.
  • 3 Comparison of Modern BNNs: Bayes by Backprop underestimates out-of-distribution variance, whereas MC Dropout typically gives greater out-of-distribution variance but unnecessarily high in-distribution variance relative to the GP.The models had limited hyperparameter tuning, particularly affecting the interpretation of MC Dropout results.
  • 3 Comparison of Modern BNNs: MC Dropout uses fewer computational resources and no additional network parameters, while Bayes by Backprop requires twice as many parameters.Bayes by Backprop may suit smooth, data-rich problems with more inference time; MC Dropout may suit large, complex, sparse-data settings with stricter time requirements.
  • 3.1 Convolutional BNNs: CNNs replace inner products with convolution-like operators, allowing multidimensional inputs and weights while retaining an equivalent MLP representation.The paper emphasizes that “convolution-like” is not mathematically identical to linear or circular convolution.
  • 3.1 Convolutional BNNs: A Bayesian convolutional neural network extends Bayes by Backprop to image classification by factorising over independent convolutional-layer weights.The BCNN was evaluated on MNIST using a Softmax likelihood and Monte Carlo integration for credible intervals.
  • 3.1 Convolutional BNNs: 98.99% BCNN test accuracy compared with 99.92% for the vanilla network on 10,000 MNIST test images, while the BCNN additionally provides uncertainty information.Credible intervals reveal greater predictive uncertainty for difficult-to-classify digits.

4 Conclusion

The review finds Bayesian methods useful for representing predictive uncertainty, but current approximate inference methods remain insufficient for reliable variance estimates. Fully factorised MFVB is tractable and often predictive-mean accurate, yet can produce over-confident predictions.

  • Limitations: Exact inference is analytically and computationally intractable for BNNs of interest, so practice relies on approximate posterior inference.The paper frames approximation as necessary for obtaining usable posterior estimates.
  • Approximate inference: Fully factorised MFVB offers a tractable lower bound and is attractive for its ease of use and accurate predictive means.It also induces an acceptable number of parameters.
  • Approximate inference: MFVB assumptions can produce over-confident predictions despite acceptable predictive performance.The survey and experiments identify this as a consequence of fully factorised approximations.
  • Bayesian CNNs: Bayesian CNNs provide credible intervals that are informative and intuitive for difficult-to-classify image examples.Their predictive performance was comparable to point-estimate CNNs on image classification tasks.
  • Future research: Further research is needed to understand modern large BNNs and achieve more accurate, scalable inference.The review specifically highlights insufficient current approximations and the need to scale exact methods.

Appendix

Figure 8 shows difficult-to-classify MNIST images from each class and their associated 95% credible intervals. Classes are ordered by true label from 0 to 9.

  • Figure 8: The figure presents difficult-to-classify MNIST images from each class.The true class labels run from 0 to 9 in alphabetical order.
  • Figure 8: The bottom plot displays 95% credible intervals for the predictions.These intervals provide the uncertainty visualization accompanying the image examples.
  • Figure 8: Compare the credible-interval displays across the class examples to inspect how predictive uncertainty varies for difficult images.The caption identifies the interval visualization but does not state an outcome of the comparison.
Loading 2006.12024v3…