PaperlayerView exact revision on arXiv

Source-linked AI summary

Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users

Laurent Valentin Jospin, Wray Buntine, Farid Boussaid, Hamid Laga, Mohammed Bennamoun

arXiv:2007.06823v3cs.LGstat.ML

TL;DR

Deep learning models can be overconfident and difficult to interpret, creating a need for principled uncertainty quantification. This tutorial surveys and organizes methods for designing, training, using, and evaluating Bayesian neural networks. It presents BNNs as promising for uncertainty-aware applications, while noting computational and approximation challenges.

  • Problem

    Deep learning models can overfit, lack explainability, and produce overconfident predictions, making reliable uncertainty quantification difficult.

  • Method

    The tutorial presents a unified workflow and literature overview for designing, implementing, training, using, and evaluating Bayesian neural networks.

  • Results

    The tutorial concludes that BNNs are a promising paradigm for applications where systems should warn when they may fail to generalize.

  • Takeaways & Limitations

    BNNs support uncertainty-aware deep learning and can also inform learning and regularization strategies for point-estimate models.

  • Takeaways & Limitations

    Exact Bayesian inference is computationally difficult for BNNs, so practical methods often use approximate or implicit Bayesian approaches.

Abstract

from arXiv · show

Modern deep learning methods constitute incredibly powerful tools to tackle a myriad of challenging problems. However, since deep learning methods operate as black boxes, the uncertainty associated with their predictions is often challenging to quantify. Bayesian statistics offer a formalism to understand and quantify the uncertainty associated with deep neural network predictions. This tutorial provides an overview of the relevant literature and a complete toolset to design, implement, train, use and evaluate Bayesian Neural Networks, i.e. Stochastic Artificial Neural Networks trained using Bayesian methods.

I. INTRODUCTION

Deep learning models can be powerful yet overfit and produce overconfident predictions, motivating Bayesian approaches that quantify uncertainty. This tutorial connects deep-learning concepts with Bayesian neural networks and presents a unified workflow for designing, training, using, and evaluating them.

  • Overfitting and overconfident predictions make uncertainty quantification important for applications such as autonomous driving, medical diagnosis, and finance.
  • Bayesian inference distinguishes aleatoric uncertainty from process noise and epistemic uncertainty from limited data through likelihood and posterior distributions.
  • Bayesian methods provide a formal framework for uncertainty quantification and for understanding regularization techniques and learning strategies in deep learning.
  • A Bayesian neural network is a stochastic artificial neural network trained using Bayesian inference, with stochastic activations or weights representing multiple possible models.
  • The tutorial presents a unified workflow for designing, implementing, training, using, and evaluating BNNs while relating Bayesian methods to traditional deep learning concepts.
  • Traditional point-estimate networks use a single parameter value and are easy to deploy, but can lack explainability and behave overconfidently on unforeseen data.

III. ADVANTAGES OF BAYESIAN METHODS FOR DEEP

Bayesian methods offer calibrated and decomposable uncertainty, explicit priors, and a framework for analyzing learning strategies. These benefits support applications including uncertainty-aware prediction and active learning, although practical methods are often approximate.

  • BNNs can provide better-calibrated uncertainty than classical neural networks, making uncertainty more consistent with observed errors.
  • BNNs distinguish epistemic uncertainty from aleatoric uncertainty, supporting data-efficient learning and high epistemic uncertainty on out-of-training-distribution points.
  • Bayesian methods make implicit learning assumptions explicit by treating priors as soft constraints analogous to regularization or data transformations.
  • The Bayesian paradigm provides a framework for interpreting regularization and other learning methods as approximately Bayesian.
  • BNNs have been applied to uncertainty quantification across fields and can support active learning by selecting unlabeled points for oracle annotation.
  • The tutorial focuses on designing stochastic models for BNNs, using probabilistic graphical models, posterior derivation, priors, and supervision choices.

A. Probabilistic graphical models

Probabilistic graphical models represent dependencies among stochastic variables and decompose their joint distributions. In Bayesian neural networks, they specify data-generation processes, posterior structure, and alternative stochasticity in weights or activations.

  • Probabilistic graphical models use graphs to represent interdependence among stochastic variables and decompose their probability distributions.
  • Directed acyclic links condition each target variable on its parents, allowing the joint distribution to be computed and the data-generation process to be defined.
  • Plate notation represents multiple independent copies of a subgraph across a batch dimension, enabling joint-probability calculations for batch variables.
  • Observed variables represent data, latent variables represent hypotheses, and their joint distribution supports posterior inference through Bayes’ formula.
  • A stochastic-weight BNN can use normal distributions for weights and biases, while its prediction may follow a normal law for regression or a categorical law for classification.
  • Stochastic-activation BNNs place probability distributions over layer activations and require accounting for dependencies across latent layers.
  • The two stochastic architectures can sometimes be made equivalent, and activation-based formulations may reduce variational parameters.

1) A good default prior:

The tutorial presents priors as design choices that shape Bayesian neural networks and connects them to regularization and alternative supervision strategies. It discusses model nonuniqueness, noisy labels, and partially labeled data as factors requiring adapted probabilistic models.

  • 1) A good default prior:: A normal prior is commonly used in practice, but the tutorial notes that no theoretical argument establishes its superiority over other formulations.Its appeal comes from mathematical properties and a simple log formulation used by learning algorithms.
  • 1) A good default prior:: Overparameterized neural networks have equivalent parameterizations, producing multimodal posteriors that can be difficult to sample and approximate.The tutorial identifies changing the functional parameterization or constraining prior support as two possible responses.
  • 1) A good default prior:: Regularization acts like a soft constraint analogous to a prior, and consistency conditions can encourage sparsity, monotonicity, or low-density decision boundaries.These conditions can be approximated from training features when the input distribution is unknown.
  • D. Degree of supervision and alternative forms of prior knowledge: Probabilistic graphical models let BNNs adapt to noisy labels, missing labels, and other supervision levels by deriving corresponding posteriors and applying inference algorithms.For noisy labels, NCAR and NAR are special cases of the more generic NNAR model; prediction can discard y and σ after posterior sampling.
  • D. Degree of supervision and alternative forms of prior knowledge: Semi-supervised Bayesian learning uses unlabeled data through data-conditioned priors or dependencies across observed and unobserved labels.The two approaches have a similar effect, with the choice depending on the preferred mathematical formulation.

2) Data augmentation:

The tutorial formulates data augmentation probabilistically by modeling augmented inputs and marginalizing them, while also extending consistency-based priors to unlabeled data. It places augmentation alongside meta-learning, transfer learning, and self-supervised learning as ways to incorporate alternative sources of prior information.

  • 2) Data augmentation:: Bayesian data augmentation models an augmented input x′ through p(x′|x, Aug) and marginalizes x′ in the posterior.The formulation is described as a probabilistic counterpart to vicinal risk.
  • 2) Data augmentation:: Monte Carlo integration approximates the augmentation integral by sampling augmentations and averaging their contributions.A single augmentation can suffice when it is resampled at every optimization iteration.
  • 2) Data augmentation:: In semi-supervised learning, priors can encourage consistent predictions between an unlabeled example and its augmentation without assigning labels to the unlabeled data.The unlabeled examples supply samples for the consistency condition rather than new annotations.
  • 3) Meta-learning, transfer learning, and self-supervised learning:: Meta-learning treats task-specific priors as conditioned on a global variable ξ, with transfer learning and self-supervised learning presented as special cases.In practice, empirical Bayes often uses a point estimate ξ̂ for this global variable.
  • 3) Meta-learning, transfer learning, and self-supervised learning:: Transfer learning reuses parameters from a main task to form a new prior, while self-supervised learning obtains labels from the data through a pretext task.Transferred parameters are selected by τ, and σ can account for additional uncertainty from domain shift.
  • 3) Meta-learning, transfer learning, and self-supervised learning:: Bayesian inference is difficult because evidence integrals and direct posterior sampling become prohibitively difficult in high-dimensional models, motivating dedicated approximate algorithms.The tutorial reviews MCMC, variational inference, simplifications, and practical comparisons of learning strategies.

A. Markov Chain Monte Carlo (MCMC)

The tutorial contrasts exact posterior sampling with scalable variational inference for Bayesian neural networks. MCMC targets exact posteriors but scales poorly, whereas variational methods optimize an approximate distribution using the ELBO and reparameterized gradients.

  • A. Markov Chain Monte Carlo (MCMC): MCMC constructs a Markov chain whose samples follow a desired distribution after burn-in, but successive samples may be autocorrelated and require storage and subsampling.These properties make generating and retaining sufficiently independent samples expensive for deep learning models.
  • A. Markov Chain Monte Carlo (MCMC): MCMC provides exact-posterior sampling but lacks scalability for BNNs, making Metropolis-Hastings more relevant than Gibbs sampling in this setting.HMC reduces rejection and correlation, while NUTS automatically tunes its hyperparameters.
  • A. Markov Chain Monte Carlo (MCMC): Metropolis-Hastings proposes θ′ from Q(θ′|θ), accepting more-likely proposals and probabilistically accepting or rejecting less-likely ones.Symmetric proposals simplify the acceptance probability, while nonsymmetric proposals require a correction term.
  • B. Variational inference: Variational inference replaces exact posterior sampling with qφ(H), whose parameters are learned by minimizing KL divergence to P(H|D).The approach scales better than MCMC but is not exact.
  • B. Variational inference: The ELBO avoids directly computing the posterior and can be optimized with stochastic variational inference using a single mini-batch per iteration.This scalability comes with slower convergence and noisy estimates when few samples evaluate the ELBO.
  • B. Variational inference: Bayes-by-backprop uses a reparameterization θ=t(ε,φ), allowing stochastic weights while preserving backpropagation through the variational parameters.The algorithm samples ε, transforms it into θ, estimates the ELBO, and updates φ by backpropagation.
  • B. Variational inference: A single-sample Bayes-by-backprop objective produces noisy gradients and substantially noisier convergence curves than classic backpropagation.Averaging the loss over multiple epochs can provide a better convergence estimate.

D. Learning the prior

Learning prior parameters jointly with variational parameters turns ELBO optimization into a tractable lower-bound objective, while scalable Bayesian inference methods adapt MCMC and variational ideas to deep learning.

  • Learning the prior: Learning prior parameters is meaningful when prior knowledge fixes most prior aspects and only a smaller parameter set is learned.This approximation is usually valid when learned prior-parameter dimensionality is much smaller than model-parameter dimensionality.
  • Learning the prior: Directly finding the optimal prior parameters is generally intractable, so variational inference optimizes the data log likelihood minus a KL-divergence term.The resulting ELBO is a lower bound on the log likelihood.
  • Learning the prior: Bayes-by-backprop can learn parametric prior and variational parameters together by sampling, backpropagating gradients, and updating both parameter sets.Algorithm 6 modifies the training loop to include the additional prior parameters.
  • Inference approaches: Monte Carlo Dropout interprets retained dropout at evaluation as variational inference, producing predictive distributions with little additional modeling effort.It is straightforward to implement, often trains faster than other variational methods, and can use an already dropout-trained model with an added prior regularizer.
  • Inference approaches: MC-Dropout is convenient and efficient but may lack expressiveness, fail to fully capture predictive uncertainty, and offer less flexibility for online or active learning.These limitations distinguish it from more flexible Bayesian methods.
  • Inference approaches: SGD dynamics can approximate posterior sampling or variational inference, but MCMC-style methods face memory costs and Laplace methods require intractable Hessian inverses for large networks.SGLD uses Langevin noise; warm restarts help explore multimodal posteriors and avoid mode collapse.

VI. SIMPLIFYING BAYESIAN NEURAL NETWORKS

The tutorial presents simplifications that reduce BNN training, evaluation, or storage costs, including stochastic last layers, Bayesian-teacher distillation, and generative compression of MCMC samples.

  • Motivation: Monte Carlo evaluation and storing many MCMC parameterizations are major practical drawbacks of BNNs.These costs motivate the mitigation strategies discussed in this section.
  • Reduced stochasticity: Using only a few stochastic layers, typically near the network output, can drastically speed training and evaluation while retaining meaningful Bayesian results.The remaining layers act as a point-estimate transformation.
  • Bayesian distillation: A non-stochastic student network can learn the marginal predictive distribution p(y|x, D) from a Bayesian teacher by minimizing a KL divergence.The teacher supplies predictive probabilities during training rather than ordinary hard labels.
  • Bayesian distillation: A larger feature-only dataset can help the student retain the teacher’s calibration and uncertainty.For classification, using BNN class probabilities instead of one-hot labels likewise preserves calibration and uncertainty.
  • Sample compression: A generative model can compress MCMC samples and generate parameterizations at evaluation time without storing the full sample set.This permits a more complex parametric distribution than those usually used in variational inference.

VII. PERFORMANCE METRICS OF BAYESIAN NEURAL NETWORKS

BNN evaluation must assess both predictive correctness and calibration because the model outputs a conditional distribution rather than a direct point estimate. Calibration curves and summary distances provide complementary diagnostics across classification and regression.

  • Evaluation goals: BNNs require separate assessment of predictive performance and calibration because they output p(y|x, D), not directly a point prediction.An optimal estimate can be extracted from the predictive distribution, but calibration must also be checked.
  • Classification calibration: Calibration curves compare observed probabilities with predicted probabilities; values below the diagonal indicate overconfidence, while values above it indicate underconfidence.A well-calibrated model has approximately equal observed and predicted probabilities.
  • Classification calibration: For multiclass classifiers, calibration can be checked independently for each class against all remaining classes.This reduces each check to a binary-classification problem.
  • Regression calibration: Regression calibration uses the normalized sum of squared residuals and a Chi-square distribution to assign predicted probabilities.The method assumes sufficiently many randomly selected inputs with independent predictions.
  • Calibration summaries: Calibration curves reveal where models are overconfident or underconfident and can support recalibration, while summary measures facilitate comparison.Examples include AUC, distance to the ideal curve, and expected calibration error.
  • Conclusion: The tutorial covers BNN design, training, and evaluation, while noting that efficient algorithms remain challenging despite the simplicity of attaching distributions to weights.It identifies potential use in applications where failures to generalize should emit warnings.

APPENDIX A IMPLEMENTING PARAMETER-EFFICIENT NORMAL

Variational covariance representations trade expressiveness for parameter efficiency. Diagonal, block-diagonal, and diagonal-plus-low-rank approximations reduce the cost of learning large covariance matrices while retaining different amounts of correlation structure.

  • Reparameterization: The reparameterization framework samples θ from qφ(θ) by transforming a random variable ε through t(ε, φ).The construction assumes the relevant probability distributions and transformation satisfy the stated bijection conditions.
  • Covariance parameterization: Exact covariance learning requires O(n^2) variational parameters and becomes computationally expensive as the parameter dimension grows.The covariance is represented through a lower-triangular matrix.
  • Diagonal approximation: A diagonal covariance approximation reduces the requirement to O(n) variational parameters.Only diagonal covariance coefficients are learned.
  • Block-diagonal approximation: A block-diagonal covariance with maximum block size w requires O(w · n) parameters but restricts learnable covariance according to parameter positions.Parameters must be close within blocks to model their covariance.
  • Diagonal-plus-low-rank approximation: A diagonal-plus-low-rank covariance uses n+r noise components and can model correlations among all components more flexibly.Its diagonal matrix D has size n × n, its lower-triangular matrix L has size n×r, and it requires O(n · r) parameters.

I. PRACTICAL EXAMPLE – BAYESIAN MNIST

The Bayesian MNIST practical example demonstrates a complete BNN workflow for digit recognition, including uncertainty behavior on seen classes, an unseen class, and white noise. Additional examples illustrate sparse-measure learning with noisy labels, structured outputs, and semi-supervised consistency strategies.

  • Bayesian MNIST: The example trains a BNN for handwritten digit recognition on MNIST while withholding digit 5 to test behavior on unseen data.The experiment compares trained classes, the withheld class, and pure white noise.
  • Bayesian MNIST: The model uses Bayesian regression with normal parameter priors, a convolutional architecture, and variational inference with an ensemble of BNNs.Weights and biases receive different prior scales, and diagonal-covariance Gaussian posteriors are used.
  • Bayesian MNIST: For seen digits, predictions are correct and confident, whereas unseen digits receive low probabilities and high uncertainty.The reported uncertainty is assessed through average predictions and standard deviations within and across samples.
  • Bayesian MNIST: White-noise inputs produce a constant average output, indicating that the network distinguishes them from characters.This is one of the three evaluation conditions used in the practical example.
  • Sparse measure: The sparse-measure example learns partially observed, noisy multivariate outputs while exploiting correlations between output functions.Its design combines a last-layer BNN with point-estimated layers and Monte-Carlo dropout.

B. Noisy labels model

The noisy-label model represents channel-specific corruption probabilistically and adds consistency conditions for structured outputs and semi-supervised learning. Its corrected consistency formulation preserves heavy-tailed behavior so outliers are not penalized too strongly.

  • Noisy labels: The model represents each output channel’s noise using a Bernoulli outlier probability and switches between σin = 0.1 and σout = 5.The formulation allows more than one noisy channel by constraining probabilities to [0, 1]^m rather than enforcing a simplex.
  • Noisy labels: A broad Gaussian prior on the unconstrained noise variable is combined with a consistency condition encouraging approximately one noisy channel.The confidence scaling factor is set to γs = 3000.
  • Noisy labels: The model learns point estimates for the noise variables and incorporates their prior-derived contribution into the training loss.This contribution is implemented as the outlierAwareSumSquareError learnable loss.
  • Consistency conditions: A second consistency condition measures distance from predictions to a known subspace and uses a normal prior with σd = 1.This encodes prior knowledge that outputs lie near a known lower-dimensional structure.
  • Consistency conditions: The corrected consistency condition retains heavy tails because the observed and prior distance-ratio distributions are reasonably matched but not exact.The correction avoids the excessive outlier penalty of the naive formulation.

E. Results

The sparse-measure results show moderate predictive accuracy and imperfect calibration, while the Paperfold case study provides a tractable setting for comparing BNN inference methods. Its architecture has two equivalent data-fitting solutions and identifiable weight-space symmetries.

  • Sparse-measure results: The sparse-measure estimator reaches an RMSE of around 0.4, with an error distribution approximating a normal distribution.Results vary across runs because the underlying function is generated randomly for each experiment.
  • Sparse-measure results: The calibration curve indicates slight underconfidence except for large outliers, where the model is overconfident.The authors attribute this pattern partly to variational inference’s inability to fit the exact posterior.
  • Paperfold: The Paperfold case study uses a small dataset and low-dimensional posterior so exact MCMC and alternative BNN training methods remain tractable.The model is nevertheless complex enough to exhibit common posterior issues.
  • Paperfold: Its two-branch feedforward model represents a planar function with a single fold along β1x + β2y + β3 = 0.One branch is linear and the other uses a RELU nonlinearity.
  • Paperfold: The model has exactly two non-equivalent data-fitting solutions, plus symmetries from swapping branches and rescaling parameters.These properties make the case study useful for examining multimodality and parameterization equivalence.

A. Comparison of training methods

The paperfold example compares MCMC, variational inference, ensembling, and ensembles of variational approximations for posterior sampling and prediction. The methods trade posterior fidelity against training and inference efficiency.

  • MCMC: MCMC provides a very good approximation of samples from the exact posterior, revealing its complex multimodal and symmetric structure.The posterior includes perpendicular, diagonal, symmetric, and hyperbolic structures associated with different parameterizations and modes.
  • MCMC: MCMC predictions fit the data with a regular function similar to a second-degree polynomial, while uncertainty increases with distance from the data region.The predicted folds remain distinct, although uncertainty persists around their exact positions.
  • Variational inference: Variational inference with a unimodal Gaussian posterior poorly approximates the exact posterior, fitting only one fold and misestimating the marginal mean and uncertainty.Its limited expressive power is especially apparent in this deliberately constructed example.
  • Ensembling: Ensembling captures multiple posterior modes and performs better than naive variational inference, but it lacks local uncertainty estimates around those modes.Training takes slightly longer because the learning procedure is restarted multiple times, while marginal inference per sample is extremely fast.
  • Variational inference + ensembling: An ensemble of variational approximations produces a Gaussian mixture that closely matches the MCMC marginal for z while retaining inference time similar to a single Gaussian approximation.It still cannot reproduce the exact posterior’s complex shapes, but demonstrates the benefits of combining variational inference with ensembling.
Loading 2007.06823v3…