PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty in Neural Networks: Approximately Bayesian Ensembling

Tim Pearce, Felix Leibfried, Alexandra Brintrup, Mohamed Zaki, Andy Neely

arXiv:1810.05546v5stat.MLcs.LG

TL;DR

Neural-network uncertainty is important but Bayesian inference is difficult to scale, while ordinary ensembling is scalable yet criticised for lacking a Bayesian basis. The paper introduces anchored ensembling, theoretically analyses its posterior approximation, and evaluates it across multiple tasks. It reports advantages over standard ensembling, competitiveness with variational methods, and state-of-the-art regression performance on 3/10 datasets.

  • Problem

    Neural-network uncertainty matters, but Bayesian inference is difficult to apply at modern parameter and data scales, while ordinary ensembling lacks a Bayesian connection.

  • Method

    The paper modifies neural-network ensembling by regularising parameters around values drawn from an anchor distribution that can equal the prior, and analyses this as randomised MAP sampling.

  • Results

    Anchored ensembling provides an advantage over standard ensembling, is competitive with variational methods, and achieves state-of-the-art regression performance on 3/10 datasets.

  • Takeaways & Limitations

    Anchored ensembling offers a scalable approximate-Bayesian approach whose theoretical exact-recovery conditions are argued to be partially present in neural networks.

  • Takeaways & Limitations

    Exact RMS design generally requires the parameter-likelihood covariance, which is infeasible for most models; the practical workaround only approximates the posterior.

Abstract

from arXiv · show

Understanding the uncertainty of a neural network's (NN) predictions is essential for many purposes. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to large numbers of parameters and data. Ensembling NNs provides an easily implementable, scalable method for uncertainty quantification, however, it has been criticised for not being Bayesian. This work proposes one modification to the usual process that we argue does result in approximate Bayesian inference; regularising parameters about values drawn from a distribution which can be set equal to the prior. A theoretical analysis of the procedure in a simplified setting suggests the recovered posterior is centred correctly but tends to have an underestimated marginal variance, and overestimated correlation. However, two conditions can lead to exact recovery. We argue that these conditions are partially present in NNs. Empirical evaluations demonstrate it has an advantage over standard ensembling, and is competitive with variational methods.

1 Introduction

Neural networks offer powerful predictions but their uncertainty is difficult to quantify at scale. The paper proposes anchored ensembling as an approximate Bayesian modification to scalable NN ensembles, with theoretical and empirical support.

  • Motivation: Uncertainty quantification is challenging for neural networks despite its importance in real-world applications and auxiliary tasks.Applications include reinforcement-learning exploration, active learning, and protection against adversarial examples.
  • Motivation: Bayesian neural-network inference is principled but often infeasible for modern models because of their large parameter and data scales.The paper specifically cites MCMC methods as examples of techniques that do not readily scale.
  • Background: Ensembling scales well by aggregating independently trained neural networks, whose predictive variance can be interpreted as uncertainty.The method retains the usual implementation of each individual network and expects similar predictions near observed data but diversity elsewhere.
  • Proposed approach: Anchored ensembling regularises parameters around values drawn from an anchor distribution that can be chosen equal to the prior, connecting the procedure to randomised MAP sampling.The paper presents this as a modification intended to align ordinary ensembling more closely with Bayesian inference.
  • Theory: Under multivariate-normal assumptions, an RMS procedure can recover the true posterior, while a practical workaround generally preserves the mean but underestimates variance and overestimates correlation.Exact recovery requires special conditions, including perfectly correlated parameters or parameters with infinite marginal likelihood variance.
  • Empirical evaluation: Experiments across regression, image classification, sentiment analysis, and reinforcement learning find anchored ensembling advantageous over standard ensembling and competitive with variational methods.The paper also argues that the exact-recovery conditions are partially present in neural networks.

2 Background

The background situates randomised MAP sampling among Bayesian and ensemble approaches, using repeated noisy MAP optimisation to approximate posterior samples. Neural-network applications are motivated by scalability, while exact transfer beyond simple settings remains difficult.

  • Bayesian inference methods: Variational inference models neural-network uncertainty explicitly or through noisy optimisation, while mean-field methods often ignore parameter correlations.Examples include explicit variational distributions, MC Dropout, Vadam, and mean-field variational inference.
  • Bayesian inference methods: Hamiltonian Monte Carlo offers gold-standard inference with limited scalability, whereas the Laplace method approximates the posterior with a multivariate normal distribution.These methods illustrate the trade-off between inference quality and computational scale.
  • Gaussian-process connection: Infinite-width Bayesian neural networks converge to Gaussian processes, which provide convenient exact predictive distributions for small problems despite superlinear data scaling.The paper uses analytical ReLU Gaussian-process predictions as ground truth when benchmarking wide neural networks.
  • Randomised MAP sampling: Randomised MAP sampling injects noise into targets or regularisation and repeatedly optimises, producing a distribution of MAP solutions that mimics posterior samples.The family includes methods called randomise-then-optimise, randomised prior functions, and ensemble sampling.
  • Randomised MAP sampling: Noise injection yields exact inference in linear regression but is difficult to transfer to neural networks and classification, where direct noise choices may not reproduce the true posterior.Wrapping optimisation in MCMC is described as more accurate but more computationally demanding.
  • RMS procedure: The exact RMS illustration samples an anchor parameter once, returns a MAP solution, and repeats this process so the distribution of solutions recovers the original posterior.Figure 2 demonstrates the procedure in a two-dimensional parameter space.

3 RMS Theoretical Results

Theoretical analysis shows that practical RMS, using an anchor distribution equal to the prior, generally recovers the posterior mean but biases covariance estimates. Exact recovery is possible under two special parameter-structure conditions.

  • Parameter-space derivation: RMS is analysed in parameter space under multivariate normal prior and parameter-likelihood assumptions, providing an abstraction of the procedure.The analysis motivates RMS beyond output-space formulations and supports its application to classification tasks.
  • Exact and practical RMS: When the parameter-likelihood covariance is known, RMS can be designed to recover the true posterior; in practice, this covariance is usually infeasible to specify.The practical workaround therefore uses only the prior distribution.
  • General-case bias: With µanc := µprior and Σanc := Σprior, the RMS approximate posterior has the correct mean but generally underestimates marginal variance.The covariance mismatch is quantified by Var[f MAP(θanc)] < Var[θ|D].
  • General-case bias: Under an isotropic prior, the RMS approximate posterior preserves the true posterior eigenvectors but generally overestimates the magnitude of parameter correlation.Thus the approximation gets the posterior orientation right while distorting covariance magnitude and dependence.
  • Special cases: Exact recovery occurs when parameters are perfectly correlated or when extrapolation parameters have infinite marginal likelihood variance.Extrapolation parameters do not affect the data likelihood but can affect new predictions; neural networks may contain such parameters through dead hidden nodes.
  • Special cases: As these two conditions are approached, RMS is expected to become increasingly accurate, and the conditions are partially present in neural networks.The exact cases are illustrated in Figure 3, while the posterior structure corresponds to mixtures of perfectly correlated and uncorrelated parameters.

4 RMS for Neural Networks

Anchored ensembling applies randomized MAP sampling to neural networks by regularizing each model around a parameter draw from the prior. Its practical posterior is generally biased in variance and correlation, but neural-network-specific conditions can enable close or exact recovery.

  • Method: Anchored ensembling regularizes neural-network parameters around independently drawn anchor values from the prior.The method replaces the prior mean in the MAP objective with an anchor draw for each ensemble member.
  • Assumptions: The practical RMS approximation is analyzed under multivariate-normal parameter-likelihood assumptions also used by other approximate Bayesian methods.Conditional likelihood plots from trained networks suggest that local modes can be approximately normally distributed.
  • Theoretical behavior: The workaround generally recovers the posterior mean but underestimates marginal variance and overestimates parameter correlation.These biases arise when only the prior distribution, rather than the parameter-likelihood covariance, is available.
  • Special cases: Exact posterior recovery can occur with perfectly correlated parameters or parameters having infinite marginal likelihood variance.The latter are called extrapolation parameters; dead hidden nodes can produce weights that affect extrapolation without affecting the observed-data likelihood.
  • Implementation: Increasing network width may encourage the strong parameter correlations required by one exact-recovery condition.Wider networks create more parameters and increase the probability of strong correlations.
  • Implementation: Small ensembles can provide useful predictive uncertainty estimates, with experiments using 5–10 networks and scaling O(MN).The required ensemble size depends more on output dimensionality than input dimensionality when networks are viewed as posterior-predictive samples.

5 Experiments

Experiments compare anchored ensembles with Bayesian approximations and standard ensembles on toy, regression, and classification tasks. Anchored ensembles capture parameter-correlation effects, approach ReLU-GP predictive distributions as width and ensemble size increase, and produce competitive or more conservative uncertainty estimates.

  • 5.1 Qualitative Tests: MFVI and MC dropout poorly capture interpolated uncertainty because their posterior approximations ignore parameter correlations.The comparison uses GP and HMC as gold-standard Bayesian references on single-layer neural networks.
  • 5.1 Qualitative Tests: Standard regularized ensembles reduce diversity by encouraging a shared solution, while unconstrained ensembles lack a prior and overfit the data.Anchored ensembles instead use loss functions corresponding to randomized MAP sampling.
  • 5.1 Qualitative Tests: KL divergence from the ReLU GP decreases as neural-network width and ensemble size increase, although a residual difference remains even for 40 networks of 1,024 nodes.This comparison is made on the Boston housing dataset.
  • 5.3 UCI Regression Benchmarks: Anchored ensembles perform best on low-noise UCI datasets, surpassing Deep Ensembles and the listed approximate inference methods.They remain competitive on other datasets, while the Deep Ensemble implementation has an advantage on higher-noise datasets through heteroskedastic modeling.
  • 5.4 Out-of-Distribution Classification: On out-of-distribution classification data, anchored ensembles generally produce the most conservative predictions, with confidence differences increasing farther from the training distribution.The experiments cover Fashion MNIST, IMDb sentiment, and CIFAR-10 using fully connected, convolutional, and embedding-plus-convolutional architectures.

6 Conclusion

The paper presents anchored ensembling as a randomized-MAP modification that approximates Bayesian inference for neural networks. Its theory identifies two exact-recovery conditions, while experiments report strong regression results and greater classification robustness than alternative ensembles.

  • 6 Conclusion: Anchored ensembling regularizes parameters around values drawn from a prior distribution to obtain approximate Bayesian inference.The procedure modifies the usual neural-network ensembling process.
  • 6 Conclusion: Perfect parameter correlations and extrapolation parameters are the two conditions shown to recover the true posterior.The paper argues that both conditions are partially present in neural networks.
  • 6 Conclusion: State-of-the-art performance was achieved on 3/10 regression datasets, while classification experiments showed greater robustness than alternative ensemble methods.The regression result outperformed popular approximate inference methods.

A Proofs

Under multivariate-normal assumptions, the analysis characterizes when randomized MAP estimates recover the Bayesian posterior and how the practical approximation deviates from it.

  • Likelihoods: The paper distinguishes data likelihood in output space from parameter likelihood in parameter space, although they return the same values for fixed data and parameters.
  • General construction: With multivariate-normal parameter likelihood and prior, the posterior is multivariate normal, and an anchor distribution can be chosen to recover it through MAP estimates.
  • Assumptions: The multivariate-normal parameter-likelihood correspondence is exact for linear regression but is only a modeling convenience for general nonlinear or non-Gaussian settings.
  • Mean recovery: Setting the anchor mean equal to the prior mean aligns the randomized MAP posterior mean with the true posterior mean.
  • Covariance recovery: Choosing the anchor covariance analytically can align the randomized MAP covariance with the true posterior covariance, but requires prior knowledge of the parameter-likelihood covariance.
  • Approximation: Using the prior covariance for the anchor generally preserves the posterior orientation but underestimates marginal variance and overestimates parameter correlations.
  • Exact cases: Exact recovery occurs for extrapolation parameters, perfectly correlated parameters, or scaled posteriors whose eigenvalues are all 0 or 1.

B.1.1 General case: Example of lemma 1.1, 1.2, theorem 2

In the general case, the approximation retains the true posterior’s orientation while shrinking marginal variances and exaggerating correlations.

  • For a general posterior, the approximation underestimates marginal variances and overestimates correlations while leaving the posterior orientation unchanged.

B.1.2 Special case: Examples of theorem 3, 4

Special parameter structures can make anchored ensembling exact or nearly exact, and a neural-network example demonstrates exact recovery for perfectly correlated final-layer weights.

  • Mixtures of parameter types: When perfectly correlated and extrapolation parameters form separate blocks, both structures are recovered exactly, while partially correlated blocks retain the general-case bias.
  • Mixtures of parameter types: Overlapping perfectly and partially correlated blocks introduce a small bias into the perfectly correlated block, without changing its correlation; the extrapolation parameter is unaffected.
  • Neural-network example: In this neural-network construction, anchored ensembling recovers the true posterior for the perfectly correlated parameters when the anchor covariance equals the prior covariance.
  • Neural-network example: A two-hidden-node ReLU network is designed so its final-layer weights are perfectly correlated because both units share the active half of the output.
  • Uncertainty-aware agents: An anchored ensemble of 5xNNs was trained with Bayesian Q-learning on discretised FetchPush, where distributional Q-values represent action uncertainty.
  • Uncertainty-aware agents: After 40,000 episodes, action distributions were narrow and separated in familiar states, narrow and overlapping after success, and evaluated for an unusual goal position.

C.3 Regression Benchmarking

Regression benchmarks compare anchored-ensemble variants and other inference methods using RMSE and NLL, while additional tests examine uncertainty and out-of-distribution behavior.

  • Regression benchmarks: Tables 3 and 4 report anchored-ensemble variants on regression benchmarks using RMSE and NLL, with discussion focused on NLL.
  • Regression benchmarks: The ReLU GP generally outperformed the ERF GP, with exceptions such as Wine, whose ordinal target favors neither method uniformly.
  • Regression benchmarks: The 10x50 anchored ensemble produced results between the 5x50 ensemble and the ReLU GP, consistent with the convergence analysis.
  • Regression benchmarks: Adding a second hidden layer improved performance over the 5x50 anchored ensemble despite minimal hyperparameter tuning.
  • Benchmark caveat: The constant-variance single 50-NN baseline sometimes performed surprisingly close to uncertainty-aware methods and even surpassed them on Energy, raising benchmark-validity questions.
  • Comparisons and OOD tests: Table 5 compares anchored ensembles with other inference methods on UCI regression log likelihood, while Table 6 reports Fashion-MNIST confidence proportions and entropy.
  • Procedure: The implementation samples anchor points from the anchor distribution, trains independently regularised networks, and combines their regression or classification estimates.

E.2 Panel of Inference Methods

The panel compares standard, regularised, anchored, Gaussian-process, HMC, and variational inference methods across synthetic regression and classification settings. Experiments vary data generation, ensemble size, and model configuration.

  • Synthetic regression: Synthetic regression uses six fixed x values from y = x sin(5x) for all methods and activation functions.The evaluated inputs are [-0.8, -0.1, 0.02, 0.2, 0.6, 0.8].
  • Synthetic regression: A second regression setting samples 20 points from [-1.5, 1.5] using y = sin(2x)+ϵ, with Gaussian noise and one shifted outlier.The noise is ϵ ∼N(0, 0.22), and the largest-x target is shifted by -0.4.
  • Inference methods: Regression comparisons include mean-square-error training, regularisation, anchoring, and a ReLU Gaussian process.These methods are assigned to sub-plots A through D.
  • Classification: Classification comparisons use cross-entropy training, regularisation, anchoring, and HMC inference on data generated with sklearn’s make blobs function.The dataset contains 30 samples, and the network uses two hidden layers with 50 units each.
  • Experimental variation: The panel varies anchored ensemble size as M = [3,5,10,20] in one experiment.The corresponding data are reused from section E.2.

E.5 KL Convergence Results

The KL convergence experiments evaluate anchored neural networks under controlled data splits, repeated runs, and increasing ensemble widths and sizes. Additional benchmark experiments cover image and text out-of-distribution settings and a stochastic FetchPush environment.

  • KL convergence: KL convergence is evaluated by training on 50% of the data and computing KL over the remaining 50%, averaged across ten runs.The ideal line uses posterior samples from the Gaussian process rather than anchored neural networks.
  • KL convergence: Boston Housing uses a 50% training split and tests on the other 50%, with hidden widths H = [4, 16, 64, 256, 1024] and ensemble sizes M = [3,5,10,20,40].The experiments use ten runs.
  • Hyperparameter selection: Hyperparameters are selected using an 80%/20% train-validation split, with Gaussian-process grid search followed by random search for anchored-network settings.Validation negative log likelihood guides selection, while larger datasets may use 2,000 training points for Gaussian-process inference.
  • Hyperparameter selection: The GP and anchored ensembles share prior and data-noise hyperparameters, while anchored-specific tuning covers batch size, learning rate, epochs, and decay rate.Double-layer networks reuse single-layer settings with a quartered learning rate and 1.5 times as many epochs.
  • Image benchmarks: Fashion MNIST experiments train a three-layer network on eight of ten classes using 48,000 examples and repeat runs with different random seeds.Out-of-distribution categories include distortions, Gaussian noise, and sparse Bernoulli noise.
  • Text benchmarks: The movie-review benchmark contains 25,000 positive or negative reviews and evaluates Reuters, repeated-word, independently randomised, and rare-word out-of-distribution inputs.Experiments are repeated five times using an embedding, one-dimensional convolutional, and hidden layer architecture.
  • Reinforcement learning: FetchPush uses sparse rewards, discretised horizontal actions with Gaussian action noise, fixed target networks updated every 500 episodes, and 40,000 simulated episodes.The final average rewards are around −0.4.
Loading 1810.05546v5…