PaperlayerView exact revision on arXiv

Source-linked AI summary

Learnable Uncertainty under Laplace Approximations

Agustinus Kristiadi, Matthias Hein, Philipp Hennig

arXiv:2010.02720v2cs.LG

TL;DR

Point-estimated neural networks can underestimate uncertainty, and Laplace approximations do not guarantee calibrated predictions. LULA adds hidden units that leave predictions unchanged while altering the loss curvature, and trains their weights to tune uncertainty. The paper reports cheap yet effective post-hoc uncertainty tuning under Laplace approximations.

  • Problem

    Point-estimated neural networks tend to underestimate predictive uncertainty, while tuning only prior precision permits limited calibration.

    Source: §1 · p. 1

  • Method

    LULA adds partly inactive hidden units to a MAP-trained network and trains their non-zero weights with an uncertainty-aware objective to alter Laplace uncertainty without changing predictions.

    Source: §1 · pp. 1–2

  • Results

    LULA provides cheap yet effective post-hoc uncertainty tuning for Laplace approximations.

    Source: §6 · p. 9

  • Takeaways & Limitations

    LULA units allow uncertainty estimates to be tuned without affecting the network's predictions.

    Source: §6 · p. 9

    The stated predictive-uncertainty proposition is scoped to real-valued networks with diagonal Laplace-approximated posteriors.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Laplace approximations are classic, computationally lightweight means for constructing Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop a formalism to explicitly "train" the uncertainty in a decoupled way to the prediction itself. To this end, we introduce uncertainty units for Laplace-approximated networks: Hidden units associated with a particular weight structure that can be added to any pre-trained, point-estimated network. Due to their weights, these units are inactive -- they do not affect the predictions. But their presence changes the geometry (in particular the Hessian) of the loss landscape, thereby affecting the network's uncertainty estimates under a Laplace approximation. We show that such units can be trained via an uncertainty-aware objective, improving standard Laplace approximations' performance in various uncertainty quantification tasks.

1 INTRODUCTION

Point-estimated neural networks can be overconfident, while Laplace approximations offer a computationally lightweight Bayesian alternative whose uncertainty may still need calibration. The paper introduces inactive hidden units that preserve predictions while changing the loss curvature, then trains their non-zero weights to improve uncertainty calibration.

1 INTRODUCTION

  • Laplace approximations provide a cost-efficient Gaussian posterior around a MAP estimate, but point-estimated networks tend to underestimate predictive uncertainty.The Gaussian covariance is based on the loss landscape near the MAP estimate.Source: §1 · p. 1
  • Prior-precision tuning offers only limited uncertainty calibration, motivating a more flexible approach that does not change point estimates.The proposed units are partly inactive because of their weight structure, so they do not affect the underlying network's predictions.Source: §1 · p. 1
  • The added units preserve network outputs while non-trivially affecting the loss curvature and Laplace-approximated uncertainty.The method adds degrees of freedom through hidden units with a particular weight structure.Source: §1 · p. 2
  • Training the units' non-zero weights with an uncertainty-aware objective is intended to improve Laplace uncertainty calibration.The schematic emphasizes better-calibrated uncertainty, particularly with respect to outliers, while retaining the same predictions.Source: §1 · p. 2, Figure 1 · p. 2

2 BACKGROUND

Bayesian neural networks replace a single point estimate with a posterior over parameters to represent epistemic uncertainty. Laplace approximations form a Gaussian around a MAP estimate, using local loss curvature to estimate uncertainty and derive predictions.

2.1 BAYESIAN NEURAL NETWORKS

  • A neural network is built from layers with weight matrices, biases, and a componentwise activation function, with each layer producing hidden units.The input is the first layer’s hidden-unit vector, and the final layer output is the network prediction.Source: §2.1 · p. 2
  • BNNs represent epistemic uncertainty by inferring a posterior over parameters rather than relying only on a MAP point estimate.In standard neural-network training, the empirical risk and regularizer correspond to a negative log-likelihood and negative log-prior.Source: §2.1 · p. 2

2.2 LAPLACE APPROXIMATIONS

  • A Laplace approximation places a Gaussian posterior around the MAP estimate, with covariance given by the inverse loss Hessian evaluated there.The Hessian is the local curvature of the loss landscape.Source: §2.2 · p. 3
  • Common Laplace practices tune prior precision post hoc, while Kronecker-factored Laplace approximates each layer’s Hessian as a Kronecker product to reduce inversion cost.The cited KFL description gives inversion cost as Θ(n^3_{l−1}).Source: §2.2 · p. 3
  • The Gaussian parameter posterior induces a Gaussian distribution over network outputs, which can be used to form predictive distributions.For binary classification, the probit approximation links output variance to predictive entropy: increasing variance moves predictive probability closer to 0.5 and increases entropy.Source: §2.2 · p. 3

3 LEARNABLE UNCERTAINTY UNITS UNDER LAPLACE APPROXIMATIONS

The paper introduces LULA units that preserve a MAP-trained network’s predictions while changing its loss curvature and Laplace uncertainty. It trains their free parameters with an uncertainty-aware objective, with practical measures to preserve the weight structure and reduce training cost.

3.1 CONSTRUCTION

  • Under network linearization, adding these units yields output variance at least as large as the original network’s for real-valued networks with diagonal Laplace posteriors.The construction’s uncertainty effect follows from non-trivial curvature along the added parameter dimensions.Source: §3.1 · pp. 4–5
  • LULA units add hidden units with a weight structure that leaves the original network’s output unchanged while allowing their free parameters to alter the loss Hessian.The resulting curvature changes affect the covariance and predictive uncertainty under a Laplace approximation.Source: §3.1 · pp. 4–5

3.2 TRAINING

  • The uncertainty-aware objective trains free LULA parameters to maintain confidence on inliers while inducing high uncertainty on outliers.The objective is task agnostic, and its first term can alternatively use the standard negative log-likelihood; omitting that term can yield maximum uncertainty everywhere.Source: §3.2 · p. 5
  • LULA training exploits parameter symmetries that preserve predictions to select parameters with loss curvature intended to improve Laplace predictive uncertainty.The objective is minimized with respect to the free parameters.Source: §3.2 · p. 5
  • The inlier set can be the dataset’s validation set, while the outlier set can be selected for the task, such as noise for regression or large-scale natural images for image classification.Source: §3.2.1 · p. 5

3.2.1 Practical Matters

  • Gradient masking keeps the added-weight structure intact during training, preserving the guarantee that LULA units do not change predictions.Gradients for parameter blocks outside the free added parameters are set to zero.Source: §3.2.1 · p. 5
  • Training uses a single-minibatch diagonal Fisher matrix as a cheaper proxy for the covariance, regardless of the Laplace variant used at test time.The paper reports this training scheme as effective and efficient.Source: §3.2.1 · pp. 5–6

4 RELATED WORK

Related work includes post-hoc Laplace uncertainty tuning and outlier-based confidence calibration. LULA is presented as a more flexible, complementary approach that does not alter prior precision.

4 RELATED WORK

  • Post-hoc Laplace tuning commonly adjusts prior precision, including by maximizing predictive log-likelihood, while outliers have also been used to improve uncertainty calibration.Earlier approaches include tuning prior precision and extending that procedure with outliers.Source: §4 · p. 6
  • LULA extends prior-precision tuning with greater flexibility and is complementary because it leaves the prior precision unchanged.Source: §4 · p. 6
  • Unlike LULA, cited outlier-based methods train non-Bayesian models and are not post-hoc.These methods include outlier regularization of maximum-likelihood training and training probabilistic models based on the Dirichlet distribution.Source: §4 · p. 6

5 EXPERIMENTS

The experiments assess LULA on toy tasks, image calibration, dataset-shift robustness, OOD detection, and computational cost. Across these settings, LULA improves uncertainty estimates while preserving the base LA’s predictions, with comparative performance varying by task and baseline.

5.1 SETUP

  • LULA is empirically evaluated on toy and image classification, UCI regression (appendix), dataset-shift robustness, and OOD experiments, using ECE, Brier score, test log-likelihood, average confidence, and FPR95.Robustness uses rotated-MNIST and corrupted-CIFAR-10; OOD uses six OOD datasets per in-distribution dataset.Source: §5 · p. 6, §5.1 · p. 7

5.2 TOY EXPERIMENTS

  • In toy regression, LULA makes uncertainty grow faster away from the data; in toy classification, it makes confidence more compact around the data without changing LA predictions.Figure 3 depicts predictive uncertainty estimates for standard and LULA-augmented LA.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3.1 Calibration

  • On SVHN, CIFAR-10, and CIFAR-100, LULA consistently improves vanilla LA calibration, often surpassing DE, but both LULA and DE calibrate worse than LA on MNIST.LULA preserves the base LA’s predictive performance; Table 1 reports ECE and test accuracy.Source: §5.3.1 · p. 7, Table 1 · p. 7

5.3.2 Dataset Shift Robustness

  • On MNIST-R, LULA improves on both vanilla LA and DE across all considered uncertainty metrics; on CIFAR-10-C it improves LA on average and is competitive with DE, marginally outperforming it at higher corruption levels.Figure 4 presents metrics as MNIST image rotation increases; Figure 5 summarizes performance by CIFAR-10-C severity.Source: §5.3.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.3.3 OOD Detection

  • LULA consistently improves the base LA on OOD confidence and detection metrics; it improves LLLA and OE as well, while DE has stronger FPR95 in general.On CIFAR-10, LULA has better OOD detection than DE; LULA is also described as cheaper to train than a single network.Source: §5.3.3 · p. 8, Table 2 · p. 9

5.4 COST

  • Training 512 LULA units on WideResNet-16-4 takes around 20 seconds, which the authors describe as negligible compared with MAP training.Table 3 reports wall-clock time for augmentation and training.Source: §5.4 · p. 8, Table 3 · p. 9

6 CONCLUSION

LULA adds partially inactive hidden units to a pre-trained MAP network so its Laplace approximation can tune uncertainty without changing predictions. The units’ parameters can be trained using inlier and outlier data, yielding cheap post-hoc uncertainty tuning.

6 CONCLUSION

  • LULA units add parameters that affect loss-landscape curvature and Laplace uncertainty without affecting network predictions.The conclusion describes this as post-hoc uncertainty tuning for Laplace approximations.Source: §6 · p. 9
  • The added units can be trained with an objective depending on both inlier and outlier datasets to calibrate predictive uncertainty.Source: §6 · p. 9

APPENDIX A PROOFS

The proofs establish that augmenting a MAP-trained network preserves its output while changing the loss geometry through additional weights. Under a diagonal Laplace approximation and network linearization, the augmented network has at least as much output variance.

APPENDIX A PROOFS

  • The augmented network’s output at the MAP estimate equals the original network’s output, establishing prediction invariance.Source: Appendix A · p. 11
  • The loss gradient with respect to the additional last-layer zero weights is nonlinear in the additional parameters.Source: Appendix A · p. 11
  • Under a diagonal Laplace approximation and the stated linearization, the augmented real-valued network’s output variance is at least the original network’s for any input.Source: Appendix A · p. 11

APPENDIX B IMPLEMENTATION

Algorithm 2 augments the network’s hidden layers with LULA units and assigns priors to their free parameters.

APPENDIX B IMPLEMENTATION

  • Algorithm 2 adds LULA units across the hidden layers, with free parameters drawn from priors.Source: Appendix B · pp. 11–12
  • The free-parameter priors may be independent Gaussians, reflecting the standard procedure for initializing neural-network parameters.Source: Appendix B · pp. 11–12

C.1 TOY DATASET

On the toy dataset, untrained LULA raises predictive uncertainty but does not adapt it well to the data; training improves calibration for inliers and outliers.

C.1 TOY DATASET

  • Untrained LULA increases predictive uncertainty, but the increase is not well-adapted to the data.Source: §C.1 · p. 12
  • Training the augmented parameters makes uncertainty more calibrated for both inliers and outliers.Source: §C.1 · p. 12

C.2 UCI REGRESSION

Across the evaluated UCI regression datasets, LULA produces higher uncertainty on outliers than the baselines while maintaining predictive performance similar to the base Laplace approximation.

C.2 UCI REGRESSION

  • The evaluation uses standardized data, 50 LULA units, 40 training epochs, 100-sample MC integration, and ten repeated train-test runs.Source: §C.2 · p. 12
  • Test-set uncertainties are generally low for all methods, with LULA slightly higher than the base Laplace approximation.Source: §C.2 · p. 12
  • LULA yields much higher uncertainty on outliers across UCI datasets than baselines while maintaining predictive performance similar to the base Laplace approximation.Source: §C.2 · p. 13, Table 5 · p. 13

C.3 IMAGE CLASSIFICATION

On CIFAR-10-C, LULA consistently improves the base Laplace approximation and is competitive with deep ensembles, especially at higher severity levels. On OOD data, it lowers confidence without introducing underconfidence on in-distribution data.

C.3 IMAGE CLASSIFICATION

  • LULA consistently improves the base LA on CIFAR-10-C and is competitive with DE, especially at higher severity levels.The comparison is reported for ECE and Brier score results.Source: §C.3 · p. 13
  • OOD detection results cover MMC, FPR95, AUROC, and AUPRC on EMNIST, KMNIST, FMNIST, LSUN, and artificial datasets.The detailed results are reported in Tables 6–9; the tables’ values are averages over five prediction runs.Source: §C.3 · p. 13, Table 6 · p. 14, Table 7 · p. 14
  • LULA lowers confidence estimates on OOD data without introducing underconfidence on in-distribution data.Source: §C.3 · p. 13

Related papers

Loading 2010.02720v2…