Source-linked AI summary
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
Yingzhen Li, Yarin Gal
TL;DR
Practical Bayesian neural-network inference needs uncertainty estimates without the severe underestimation associated with dropout VI, while existing alpha-divergence methods are difficult to use with real models. The paper re-parametrises alpha-divergence objectives for dropout, requiring only a changed loss and stochastic forward passes during training. The resulting method improves uncertainty estimates and accuracy over VI, and its epistemic uncertainty helps distinguish adversarial from non-adversarial images.
Problem
Dropout VI is practical but can severely underestimate model uncertainty, while existing alpha-divergence techniques are restricted to Gaussian approximations and require radical model changes.
Method
The paper re-parametrises alpha-divergence objectives into a dropout-compatible inference technique implemented by changing the neural-network loss and using stochastic forward passes during training.
Results
Specific alpha choices improve uncertainty estimates and accuracy compared with VI in dropout BNNs without slowing convergence, and uncertainty increases on adversarial MNIST images.
Takeaways & Limitations
Examining epistemic model uncertainty can distinguish adversarial images from non-adversarial images within the demonstrated setting.
Takeaways & Limitations
The adversarial-input experiment is preliminary, and the authors state that more research is needed to address difficulties with adversarial inputs.
Abstract
from arXiv · showhide
To obtain uncertainty estimates with real-world Bayesian deep learning models, practical inference approximations are needed. Dropout variational inference (VI) for example has been used for machine vision and medical applications, but VI can severely underestimates model uncertainty. Alpha-divergences are alternative divergences to VI's KL objective, which are able to avoid VI's uncertainty underestimation. But these are hard to use in practice: existing techniques can only use Gaussian approximating distributions, and require existing models to be changed radically, thus are of limited use for practitioners. We propose a re-parametrisation of the alpha-divergence objectives, deriving a simple inference technique which, together with dropout, can be easily implemented with existing models by simply changing the loss of the model. We demonstrate improved uncertainty estimates and accuracy compared to VI in dropout networks. We study our model's epistemic uncertainty far away from the data using adversarial images, showing that these can be distinguished from non-adversarial images by examining our model's uncertainty.
1. Introduction
Bayesian deep learning preserves uncertainty information that deterministic deep learning often loses, but practical dropout VI can underestimate uncertainty. The paper proposes a dropout-compatible alpha-divergence method implemented by changing the loss, and reports improved uncertainty estimates and accuracy compared with VI.
- Bayesian models capture parameter uncertainty and can express increased uncertainty on examples outside the data distribution.
- Dropout VI provides practical approximate inference through stochastic forward passes, but can severely underestimate model uncertainty.Dropout layers are added before weight layers, and Monte Carlo integration is performed at test time.
- Existing alpha-divergence methods are difficult to use because they rely on Gaussian approximations, explicit density evaluations, and radical model-code changes.
- The proposed re-parametrisation derives a simple dropout inference technique requiring only a changed neural-network loss and multiple stochastic training passes.The method uses dropout approximate variational distributions and stochastic network outputs.
- Specific alpha choices improve uncertainty estimates and accuracy over VI without slowing convergence, while adversarial-image uncertainty can distinguish adversarial from non-adversarial images.
2. Background
Bayesian neural networks place distributions over neural-network weights and infer a posterior from observed data. Approximate inference uses variational distributions, with dropout providing a practical stochastic approximation whose uncertainty can nevertheless be underestimated by the VI objective.
- Bayesian neural networks place prior probability distributions over network weights and infer their posterior using observed inputs and outputs.
- Approximate inference seeks a tractable distribution over weights that generated the data because the exact posterior is generally intractable.
- Variational inference defines an approximating distribution qθ(ω) and minimizes its KL divergence from the posterior.
- Dropout can be interpreted as variational Bayesian inference by applying random multiplicative noise to neural-network weight matrices.Its approximate weight distribution can be represented by a mixture of two Gaussians, with one mean fixed at zero.
- Dropout VI underestimates model uncertainty because exclusive KL penalizes probability mass in unsupported regions but not missing mass where the posterior has support.
3. Black-box α-divergence minimisation
Black-box alpha-divergence methods generalize approximate inference beyond VI but are difficult to scale and implement with flexible approximations. Their Monte Carlo formulation introduces computational and estimation issues, motivating an improved energy function for dropout.
- Alpha-divergence methods form a family of approximate inference objectives related to power expectation propagation, including VI and EP as special cases.
- Power-EP maintains factors for every likelihood term, creating O(N) posterior-approximation space complexity, whereas BB-alpha reduces this overhead to O(1).
- BB-alpha has been applied to Bayesian neural networks and often performs better with nonzero alpha than with the VI setting.Prior work reports applications in regression, classification, and model-based reinforcement learning.
- Existing BB-alpha energy functions are difficult to extend to dropout because they assume exponential-family approximations, offer limited intuitive interpretation, and can impose prohibitive computation.
- The naive dropout implementation would collect M K masked weight-matrix samples for a mini-batch of M datapoints and K samples, increasing computational burden.
- The paper motivates an improved BB-alpha energy whose additive variational-free-energy structure avoids repeated evaluation of the approximation density.The improved formulation is intended to support dropout and other flexible approximation structures.
4. A New Reparameterisation of BB-α Energy
The paper reparameterizes the BB-α energy into a simpler objective compatible with dropout approximate posteriors, requiring stochastic forward passes and loss modification. The formulation connects α choices to predictive-likelihood or VI-style training behavior and applies to classification and regression.
- Reparameterisation of BB-α Energy: The reparameterization uses cavity distributions to reduce BB-α computational overhead while retaining a continuous dependence on α.As α/N approaches zero, the approximate posterior and energy converge toward their VI counterparts under the stated finiteness assumptions.
- Objective Function: The proposed objective replaces the likelihood with a loss-based expression and approximates expectations using Monte Carlo samples from the approximate posterior.The resulting objective uses log-sum-exp over K sampled parameter values and also approximates the marginal likelihood.
- Dropout BB-α: α = 1 directly optimizes predictive log-likelihood, whereas α → 0 recovers the VI training-accuracy objective.Intermediate choices such as the Hellinger value balance training error reduction and predictive-likelihood improvement.
- Dropout BB-α: For dropout inference, implementation requires multiple stochastic forward passes through the network and a modified loss function.Each pass samples a new dropout mask, producing stochastic outputs that are combined by the induced BB-α loss.
- Dropout BB-α: In classification, stochastic network probabilities are raised to power α, averaged, and passed into cross entropy with L2 regularization from the approximate KL term.The induced classification loss can be implemented in Keras with a short code snippet.
- Dropout BB-α: The same approach extends to regression by combining multiple stochastic outputs within a dropout BB-α loss based on the squared-error likelihood.The regression implementation resembles a standard L2 objective while using repeated stochastic network evaluations.
5. Experiments
The experiments evaluate reparameterised BB-α with dropout across regression, classification, adversarial attacks, and runtime trade-offs. Nonzero α often improves uncertainty-related metrics and robustness, while uncertainty helps distinguish adversarial inputs.
- The study evaluates reparameterised BB-α with dropout on regression, classification, adversarial attacks, and training-time sampling trade-offs.The experiments compare α values across fully connected and convolutional networks and assess uncertainty on adversarial MNIST examples.
- 5.1. Regression: Non-VI α values significantly improve test log likelihood on almost all selected regression datasets while maintaining comparable test error.α = 0.5 gives overall good test LL and RMSE results.
- 5.1. Regression: The best dropout model outperforms Gaussian VI on test log likelihood across almost all regression datasets and is on par with HMC or a sparse Gaussian process on some datasets.HMC is described as the current gold standard for Bayesian neural networks, while the Gaussian process is known to be strong in regression.
- 5.3. Detecting Adversarial Examples: Dropout Bayesian networks are more robust to untargeted and targeted attacks than the deterministic neural network, especially for α = 1.0 in the untargeted attack.Their predictive uncertainty can distinguish adversarial examples from ordinary test inputs in these experiments.
- 5.4. Run time trade-off: The runtime experiment varies training-time samples K ∈ {1, 10, 100} while tracking MNIST test accuracy and test log likelihood against wall-clock time.K = 1 reduces to standard VI, whereas larger K values address estimator bias with additional sampling cost.
6. Conclusions
The paper presents a practical BB-α extension for dropout approximating distributions that is easy to implement and often outperforms existing approximate inference techniques.
- The proposed technique extends the BB-α objective to dropout approximating distributions.The authors provide an implementation code snippet for the induced loss in the appendix.
A. Code Example
The code example implements BB-α inference by defining a loss over Monte Carlo dropout logits and generating multiple stochastic network outputs.
- The classification implementation defines a loss that accepts Monte Carlo sampled logits.Regression can be implemented similarly.
- The BB-α loss computes log-softmax values, Monte Carlo log-likelihoods, and a log-sum-exp aggregation scaled by α.The implementation returns the negative scaled log-sum-exp objective.
- Monte Carlo samples are generated by repeatedly applying initialized Keras layers to the input.The default generator creates K_mc stochastic outputs and packs them into a batch-by-sample-by-class tensor.
- The example uses auxiliary functions to apply layers sequentially and compute a numerically stable log-sum-exp operation.The layer application iterates through the supplied layer list.
B. Alpha-divergence minimisation
The paper uses alpha-divergence objectives as alternatives to variational inference, focusing mainly on Amari’s and Rényi’s definitions and their relationship to power EP.
- The work mainly uses Amari’s alpha-divergence adapted to EP and Rényi divergence from information theory.The two definitions can be converted into each other.
- Power EP minimizes alpha-divergence through projection-based updates when the approximating posterior belongs to an exponential family.The update uses moment matching to a tilted distribution.
- The tilted distribution is proportional to p(ω)^αq(ω)^(1−α).This distribution combines the target distribution and approximation through α-dependent powers.
C. Original Derivation of BB-α Energy
The original BB-α derivation connects power EP energy with exponential-family approximations, while sharing local parameters to reduce memory overhead and yield a global optimization problem.
- The original BB-α energy is formulated for approximating distributions based on exponential-family parameters.The formulation begins by specifying an exponential-family prior and approximate distribution.
- Expectation propagation approximates each non-conjugate factor with a simpler exponential-family factor before constructing the approximate distribution.The resulting local parameters are updated through cavity, tilted-distribution, projection, and site-recovery steps.
- At convergence, the local parameters return a fixed point of the power EP energy.The local parameters are maintained during optimization before convergence.
- Power EP requires O(Nd^2) space for Gaussian approximations because it stores local parameters for all N data points.This becomes prohibitive for large neural-network datasets.
- BB-α shares the local parameters by setting λ_n = λ, producing a global optimization problem in which only λ_q is optimized.Under a regular exponential-family condition, λ_q and λ have a one-to-one mapping for fixed λ_0.
- The original BB-α formulation is difficult for practitioners to interpret, and its derivation from power EP is ad hoc and lacks theoretical justification.Tying Lagrange multipliers changes the primal problem and loses some guarantees, although the approximation has worked in real-world settings.
D. Full Regression Results
The full regression results report average negative test log likelihood in nats and average test RMSE.
- Average negative test log likelihood is reported in nats for the regression experiment.
- Together, the tables provide two complementary metrics for evaluating the regression experiment.
- Average test RMSE is reported as a second regression metric.