Source-linked AI summary
'In-Between' Uncertainty in Bayesian Neural Networks
Andrew Y. K. Foong, Yingzhen Li, José Miguel Hernández-Lobato, Richard E. Turner
TL;DR
Mean-field variational inference can miss calibrated uncertainty between separated observation clusters, while standard UCI splits do not adequately test that setting. The paper compares approximate inference methods on standard and custom gap splits, finding that linearised Laplace handles in-between uncertainty better than MFVI. This highlights the importance of how posterior approximations affect uncertainty expressed in function space.
Problem
Standard UCI splits do not adequately test whether neural networks express calibrated uncertainty between separated observation clusters, a capability relevant to active learning and critical decision-making.
Source: §1 · p. 1Method
Results
MFVI fails to provide calibrated in-between uncertainty, whereas the linearised Laplace approximation performs far better in this regard.
Source: §4 · p. 3Takeaways & Limitations
The findings underscore the need to consider how posterior approximations in parameter space affect the expressiveness of uncertainty estimates in function space.
Source: §4 · p. 3The methods assume a diagonal Gaussian prior over weights and biases, with a prior variance for each parameter.
Source: §2 · pp. 1–2
Abstract
from arXiv · showhide
We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean-field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such overconfidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle 'in-between' uncertainty much better for small network architectures.
1. Introduction
Reliable uncertainty matters for neural networks used in decision-making, but MFVI can be poorly calibrated between separated observation clusters. The paper introduces a gap-based UCI evaluation and finds that linearised Laplace handles this uncertainty better than MFVI.
1. Introduction
- Reliable uncertainty estimates alongside standard networks’ computational and statistical efficiency would support active learning, reinforcement learning, and critical decision-making.Neural networks trained by maximum likelihood or MAP are known to underestimate uncertainty.Source: §1 · p. 1
- MFVI is widely used and gives state-of-the-art performance in nonlinear regression, with reparameterised MFVI often outperforming several approximate-inference and ensemble baselines on UCI regression.It is competitive with Monte Carlo Dropout on that benchmark.Source: §1 · p. 1
- MFVI can fail catastrophically on gap-based UCI splits despite performing well on standard splits, while linearised Laplace performs well on both.The standard UCI test set is uniformly subsampled, whereas the new benchmark tests on the middle region of the dataset.Source: §1 · p. 1
- Well-calibrated predictions should be confident near observation clusters and uncertain far from data and between separated clusters.The standard UCI benchmark does not adequately test this in-between uncertainty because its test set is uniformly subsampled.Source: §1 · p. 1
2. Methods
The methods compare variational inference and Laplace approximation for Bayesian neural networks with Gaussian outputs. They contrast mean-field and full-covariance variational posteriors with sampled and linearised Laplace prediction.
2. Methods
- The networks interpret their outputs as Gaussian means with homoscedastic observation noise and place a diagonal Gaussian prior over weights and biases.The prior variance is specified for each parameter.Source: §2 · pp. 1–2
- Variational inference learns an approximate posterior by optimizing a Monte Carlo estimate of the ELBO, which minimizes its KL divergence from the true posterior, then predicts by sampling from that approximation.Source: §2 · p. 2
- MFVI uses a fully factorized Gaussian posterior, while FCVI uses a more flexible full-covariance Gaussian with a number of variational parameters quadratic in the network parameter count.Source: §2 · p. 2
- The Laplace approximation centers a Gaussian posterior at the MAP solution and sets its covariance to match posterior curvature, using a positive-semidefinite Gauss-Newton matrix evaluated with first derivatives.The Gauss-Newton calculation requires one backward pass per dataset element; covariance-matrix inversion scales cubically in the number of parameters and is tractable for the smaller UCI regression networks considered.Source: §2 · p. 2
- Sampled Laplace predicts by Monte Carlo sampling from the approximate posterior, whereas Linearised Laplace linearizes the network output at the MAP solution to obtain an exactly solvable linear Gaussian predictive model.The paper notes that Laplace sampling can cause severe underfitting and says linearisation empirically alleviates it.Source: §2 · p. 2
3. Experiments
Experiments show MFVI misses increased uncertainty between separated input clusters, while LL and HMC capture it in a synthetic setting; on UCI gap splits, only LL avoids catastrophic outcomes.
Synthetic 1D Dataset
- With two separated input clusters, MFVI has similar-sized error bars in data and gap regions, while LL and HMC show increased uncertainty between clusters.FCVI increases uncertainty in the middle but is slightly underconfident in the data region.Source: §3 · p. 2
- A single-hidden-layer ReLU BNN with deterministic input weights and mean-field output weights must have input-convex output variance, limiting uncertainty increases between low-uncertainty regions.The authors note this result does not immediately apply to tanh activations and mean-field input weights, but illustrates that mean-field can restrict function-space uncertainty complexity.Source: §3 · pp. 2–3
- MFVI cannot fit outer-region data while increasing uncertainty between regions unless its approximate posterior captures strong dependencies among parameters.In a mean-field distribution, parameter uncertainty that increases uncertainty in the gap instead causes uncontrolled variation in the data-region fit.Source: §3 · p. 3
UCI Regression Datasets
- The UCI gap splits hold out the middle third of points along each input dimension, using the outer two-thirds for training and 10% of that training set for validation.Each dataset therefore has one split per input dimension; the experiment checks whether methods fit standard splits while avoiding catastrophic gap-split results.Source: §3 · p. 3
- On UCI gap splits, LL is the only method that performs well on standard splits without catastrophic results on gap splits.MAP and MFVI perform similarly poorly on energy and naval, where increased in-between uncertainty is needed to avoid overconfident extrapolations.Source: §3 · p. 3
4. Conclusions
The paper concludes that MFVI misses calibrated in-between uncertainty and standard UCI splits do not adequately test it; linearised Laplace performs better, though it is less scalable.
4. Conclusions
- MFVI fails to provide calibrated in-between uncertainty, and standard UCI splits do not adequately test for it; the linearised Laplace approximation performs far better in this regard.The authors contrast mean-field's loss of uncertainty expressiveness with the less scalable linearised Laplace approximation.Source: §4 · p. 3
- The authors identify studying how parameter-space posterior approximations affect function-space uncertainty as a key direction for approximate inference.Future work will investigate conditions an approximate posterior must satisfy to capture in-between uncertainty reliably.Source: §4 · p. 3
- The authors propose investigating whether combining K-FAC Laplace with linearisation improves results.This is presented as a future-work direction.Source: §4 · p. 3
- Figures 2 and 3 report average test log-likelihoods on standard and gap splits, respectively, for one- and two-hidden-layer BNNs.Each layer has 50 hidden units.Source: Figure 2 · p. 4, Figure 3 · p. 4
A. Linearised Laplace Approximation
Linearised Laplace approximates the network output around θMAP, then uses standard linear-Gaussian formulas to obtain a predictive distribution.
A. Linearised Laplace Approximation
- Linearised Laplace approximates the network output around θMAP and uses standard linear-Gaussian formulas to obtain the predictive distribution.The passage introduces the approximating distributions between linearisation and the use of those formulas.Source: A. Linearised Laplace Approximation · p. 6
B. Convex Variance Result
For the specified mean-field output-weight posterior, the predictive variance is convex under ReLU conditions, limiting its ability to express greater uncertainty between low-uncertainty regions. The paper tests this restriction on a 1D dataset.
B. Convex Variance Result
- The analysis assumes a fully factorized distribution over output weights and deterministic input weights and biases; output variance is treated as uncertainty at x.Source: B. Convex Variance Result · p. 6
- On 1D ReLU experiments, MFVI and MFVI-output have convex variance, while BLR's full-covariance posterior shows increased in-between uncertainty.The comparison uses maximum-likelihood input parameters for MFVI-output and BLR.Source: Figure 4 · p. 6, B. Convex Variance Result · p. 6
- With ReLU activations, the mean-field output posterior's predictive variance is convex, so it cannot be higher between two low-uncertainty regions.The argument requires the stated condition on the activation and its derivatives.Source: B. Convex Variance Result · p. 6
C. Analysis of Uncertainty in Toy Case
In a two-point toy setting, fitting both data regions while expressing greater uncertainty between them requires strong parameter dependencies. Mean-field independence prevents that coordination, while HMC samples illustrate how varying kink positions can yield larger in-between uncertainty.
C. Analysis of Uncertainty in Toy Case
- A smoothing argument establishes the ReLU convex-variance result by approximating the kink and taking a pointwise limit of convex variances.Source: C. Analysis of Uncertainty in Toy Case · p. 6
- HMC samples shift kink positions and thereby yield larger uncertainty between the two observations.The figure notes that some samples have only one kink between x1 and x2.Source: Figure 5 · p. 7
- A reasonably broad prior makes kink locations roughly uniform over the interval, allowing positions between the observations to fit the data equally well.Source: C. Analysis of Uncertainty in Toy Case · p. 7
- In the two-point toy setting, expressing large uncertainty between low-uncertainty data regions while retaining their fit requires strong parameter dependencies.The construction uses a two-hidden-unit ReLU network and observations at x1 and x2.Source: C. Analysis of Uncertainty in Toy Case · p. 7
- MFVI assumes parameter independence, so added randomness disrupts data-region outputs; it either harms fit or minimizes variances, and in practice prunes hidden units.The minimized variances incur a large penalty in the KL term in the ELBO.Source: C. Analysis of Uncertainty in Toy Case · p. 7
D. Extra Results and Experimental Details
The extra results compare MFVI and linearised Laplace on standard and gap splits, and add ReLU synthetic-data diagnostics. They also document UCI experimental settings and clarify the scope of the reported comparisons.
D. Extra Results and Experimental Details
- For the synthetic 1D experiment, a ReLU BNN and an infinite-width ReLU-kernel GP are included; ReLU makes linearised Laplace uncertainty discontinuous through its input gradient.The plotted predictive distribution omits output noise.Source: D. Extra Results and Experimental… · p. 7, Figure 6 · p. 8
- Training uses ADAM at learning rate 0.001 for 20,000 epochs, while UCI hyperparameters and training schedules vary by method and dataset size.The experiments also describe priors, HMC sampling settings, and posterior-sample estimates for test log-likelihoods and error bars.Source: D. Extra Results and Experimental… · pp. 7–8
- Duvenaud and Adams (2015) is described as an apparent in-between-uncertainty exception because its radial-basis nonlinearities have only local effects.The paper says its argument does not apply to that case.Source: D. Extra Results and Experimental… · p. 7
- The UCI results use 50-neuron hidden layers, normalized inputs and outputs, and hyperparameter selection on a 10% training-set validation split.The selected hyperparameters are used to retrain with the validation set combined into training data, repeated for each split.Source: D. Extra Results and Experimental… · p. 8
- Figure 7 compares MFVI-ReLU with linearised Laplace tanh on standard splits, with positive differences indicating better performance by Laplace.The pairwise comparisons use test log-likelihoods across datasets.Source: Figure 7 · p. 8, D. Extra Results and Experimental… · p. 8
- Tables 1 and 2 report average test log-likelihoods for standard and gap splits, respectively, with best results within one standard error in bold.The accompanying pairwise plots show test log-likelihood comparisons across datasets.Source: D. Extra Results and Experimental… · p. 8, Table 1 · p. 9, Table 2 · p. 9
- On gap splits, MFVI fails catastrophically on energy and naval, while the methods perform comparably when those datasets are removed.Figure 8 covers the gap splits; Figure 9 repeats the comparison without energy and naval.Source: Figure 8 · p. 9, Figure 9 · p. 9