Source-linked AI summary
Bayesian Optimization Meets Laplace Approximation for Robotic Introspection
Matthias Humt, Jongseok Lee, Rudolph Triebel
TL;DR
Robotic deep learning needs reliable confidence estimates, but available DNN uncertainty methods trade fidelity against scalability. This paper applies scalable Laplace Approximation and Bayesian Optimization to address its under-fitting, reporting fewer iterations than random search and scalability to large datasets and architectures.
Problem
DNN uncertainty methods face a fidelity–scalability trade-off, despite reliable confidence estimates being relevant to robotic long-term autonomy.
Source: §I · p. 1Method
The paper estimates DNN weight uncertainty with scalable Laplace Approximation and applies Bayesian Optimization to address Laplace Approximation’s under-fitting.
Source: §I · pp. 1–2Results
The proposed Bayesian Optimization approach requires fewer iterations than random search, and the framework scales to large network architectures and datasets.
Source: §I · p. 2Takeaways & Limitations
The authors present Laplace Approximation with Bayesian Optimization as a practical approach to model-uncertainty estimation, demonstrating scalability on ImageNet with a large DNN.
Source: §V · p. 5Laplace Approximation assumes a bell-shaped posterior and can under-fit when the true posterior is asymmetric or the DNN Hessian is undetermined.
Source: §I · p. 2
Abstract
from arXiv · showhide
In robotics, deep learning (DL) methods are used more and more widely, but their general inability to provide reliable confidence estimates will ultimately lead to fragile and unreliable systems. This impedes the potential deployments of DL methods for long-term autonomy. Therefore, in this paper we introduce a scalable Laplace Approximation (LA) technique to make Deep Neural Networks (DNNs) more introspective, i.e. to enable them to provide accurate assessments of their failure probability for unseen test data. In particular, we propose a novel Bayesian Optimization (BO) algorithm to mitigate their tendency of under-fitting the true weight posterior, so that both the calibration and the accuracy of the predictions can be simultaneously optimized. We demonstrate empirically that the proposed BO approach requires fewer iterations for this when compared to random search, and we show that the proposed framework can be scaled up to large datasets and architectures.
I. INTRODUCTION
The paper applies scalable Laplace Approximation to estimate DNN uncertainty for robotic introspection, then uses Bayesian Optimization to address Laplace Approximation’s under-fitting. It reports fewer optimization iterations than random search and scalability to large architectures and datasets.
I. INTRODUCTION
- Reliable confidence estimates matter for long-term autonomous robots because assessed reliability can inform planning and mission management toward higher mission success.The paper frames introspection as assessing failures and the quality of internal states such as sensor data, predictions, and world representations.Source: §I · p. 1
- DNN uncertainty methods face a fidelity–scalability trade-off: scalable MC-dropout can underestimate uncertainty, while HMC is treated as a gold standard that does not scale to modern architectures and large datasets.The paper motivates a first-line method balancing fidelity and scalability for robotic applications.Source: §I · p. 1
- Laplace Approximation models the DNN weight posterior with a Gaussian around the MAP estimate, using a scaled inverse Hessian for covariance.Kronecker factorization can model more expressive posterior families at large scale than Bernoulli or diagonal covariance approximations.Source: §I · p. 1, Figure 1 · p. 1
- The proposed Bayesian Optimization technique addresses Laplace Approximation’s under-fitting and empirically requires fewer iterations than random search.The paper also proposes scalable Laplace Approximation for robotic introspection.Source: §I · p. 2
- The authors identify two sources of Laplace Approximation under-fitting: a bell-shaped posterior assumption that cannot represent asymmetry and an undetermined Hessian from DNN over-parameterization.They propose Bayesian Optimization to address this under-fitting.Source: §I · p. 2
- The framework is reported to scale to large network architectures and datasets.The paper presents this as an empirical contribution alongside its comparison with random search.Source: §I · p. 2
II. RELATED WORK
The related work contrasts Bayesian uncertainty methods and robotic introspection approaches, positioning Laplace Approximation as a simple deterministic alternative and the paper’s Bayesian approach as task-flexible.
II. RELATED WORK
- MC-dropout requires dropout layers and has a limited posterior family, while sampling and variational inference methods are described as often unscalable or fragile; the authors build on simpler, deterministic Laplace Approximation.They note that modern networks often use batch normalization rather than dropout layers.Source: §II · p. 2
- Robotic introspection research includes learned failure predictors trained on failure-regime data and unsupervised novelty detection for non-stationary or open-set conditions.Prior work also used confidence estimates for decision-making and active learning,.Source: §II · p. 2
- The authors’ Bayesian approach is not tied to one task and may also apply to pruning, active learning, continual learning, and second-order DNN optimization.They characterize ensembles and probabilistic programming as orthogonal alternatives.Source: §II · p. 2
A. Laplace Approximation
Laplace Approximation (LA) models the DNN weight posterior with a Gaussian centered at the trained weights, enabling approximate Bayesian prediction by sampling. Its covariance relies on curvature information, and regularization addresses potential problems with the approximation.
A. Laplace Approximation
- LA approximates the weight posterior with a Gaussian around the mode, using a second-order Taylor expansion of the log posterior.The Gaussian form follows by exponentiating the quadratic approximation involving the average Hessian of the negative log posterior.Source: A. Laplace Approximation · p. 2
- Posterior weight samples support approximate Bayesian prediction for an unseen input through Monte Carlo integration.The Hessian is generally too large to compute, store, and invert, so curvature matrices may use diagonal, block-diagonal, or Kronecker-factored approximations.Source: A. Laplace Approximation · pp. 2–3
- The approximation can fail when curvature is not positive semidefinite, overestimates variance, or assigns mass to low-probability regions of the true posterior.These issues arise in the curvature and Gaussian posterior approximations.Source: A. Laplace Approximation · p. 3
- Regularization sets covariance to a scaled inverse of the Hessian, with τ interpretable as prior precision and N as dataset size or tunable hyperparameters.A single set of regularization parameters is used across curvature matrices for the DNN layers.Source: A. Laplace Approximation · p. 3
B. Bayesian Optimization
Bayesian Optimization (BO) searches for a minimum of an expensive, potentially noisy black-box objective by using a probabilistic model to select evaluations. This differs from grid or random search, and applies to costly DNN hyperparameter evaluation.
B. Bayesian Optimization
- BO models an expensive, potentially noisy black-box objective and selects candidates by minimizing a utility based on the model’s posterior.The model is updated alongside optimization, and the selected candidate is then evaluated on the objective.Source: B. Bayesian Optimization · p. 3
- Expected Improvement and Lower Confidence Bound are common utilities, while Gaussian Processes are a common model; grid and random search instead predefine or randomly sample candidates.Source: B. Bayesian Optimization · p. 3
- DNN hyperparameter tuning is an expensive black-box objective because candidate performance may require hours, days, or weeks of training to evaluate.Source: B. Bayesian Optimization · p. 3
C. Bayesian Optimization meets Laplace Approximation
The paper applies BO to tune LA regularization parameters, whose evaluation is costly and noisy and whose values affect both accuracy and probabilistic output. A GP with Hedge selects candidates, while accuracy, NLL, and ECE provide performance measures.
C. Bayesian Optimization meets Laplace Approximation
- BO searches LA regularization parameters because each evaluation requires repeated network passes and stochastic sampling, making it computationally burdensome and noisy.Larger sample counts yield more accurate approximations, while excessive regularization makes probabilistic outputs deterministic and too little hurts accuracy.Source: C. Bayesian Optimization meets Laplace… · p. 3
- A Gaussian Process with Hedge selects among utility functions to search for one or multiple sets of LA regularization parameters.The utilities are optimized independently to propose candidates, and the best proposal is chosen using weights updated according to metric-derived costs; β is a Hedge hyperparameter.Source: C. Bayesian Optimization meets Laplace… · pp. 3–4
- Accuracy and negative log-likelihood are insufficient alone for reliable uncertainty assessment, so the method also uses expected calibration error.ECE is the binned weighted difference between accuracy and confidence; NLL can increase even as classification accuracy improves.Source: C. Bayesian Optimization meets Laplace… · p. 4
IV. EXPERIMENTS
The experiments assess BO’s effectiveness, efficiency, and scalability for LA through a small-scale CIFAR-10 comparison with random search and a large-scale ImageNet experiment.
IV. EXPERIMENTS
- The evaluation compares BO with random search on CIFAR-10 and tests scalability on ImageNet using publicly available pretrained networks.The authors state that the experiments use PyTorch in Python and that the code is publicly available.Source: §IV · p. 4
A. Small scale experiment
In the ResNet18/CIFAR-10 experiments, Bayesian optimization (BO) reached better Laplace Approximation (LA) performance faster than random search (RS), including when the search used separate regularization parameters for different layers. LA with BO improved calibration and slightly improved accuracy over the deterministic network.
A. Small scale experiment
- The two-hyperparameter experiment used a pre-trained ResNet18 with 93.32% validation accuracy and optimized a validation cost combining classification error and ECE.The baseline ECE was 4.57%, and the baseline cost was 11.25.Source: A. Small scale experiment · p. 4, Figure 3 · p. 4
- 7 evaluations (versus 13 for RS) let BO surpass the standard DNN, and BO found a marginally superior final parameter set.The comparison used one set of LA hyperparameters.Source: A. Small scale experiment · p. 4, Figure 3 · p. 4
- BO also converged faster than RS when four hyperparameters separately regularized convolutional and final fully connected layers.The separate-layer setup was expected to help because curvature magnitudes differed across layers.Source: A. Small scale experiment · p. 4, Figure 5 · p. 5
- The reported method significantly improved ResNet18 performance on CIFAR-10, with a slight accuracy improvement and a major reduction in miscalibration, though the change might not be immediately visible in preceding figures.Source: A. Small scale experiment · p. 5
B. Large scale experiment
On ImageNet with DenseNet121, BO and RS both found LA regularization parameters that outperformed the deterministic network. Within the fixed evaluation budget, BO achieved lower ECE and slightly higher accuracy than RS.
B. Large scale experiment
- The ImageNet experiment used pre-trained DenseNet121 and reserved half of the official validation set for LA hyperparameter optimization.The deterministic baseline had 74.42% accuracy and 2.63% ECE on the other half.Source: B. Large scale experiment · p. 5
- The reliability diagrams show accuracy by confidence bin and the calibration gap; BO's clearest gain was lower overconfidence in the 0.6–0.7 confidence bin.The deterministic network was already relatively well calibrated, with ECE of 2.63%.Source: B. Large scale experiment · p. 5, Figure 6 · p. 5
- Within the 10-evaluation budget, BO achieved lower ECE and slightly higher accuracy than RS for ImageNet LA.Both optimization methods outperformed the deterministic network.Source: B. Large scale experiment · p. 5, Figure 6 · p. 5
V. CONCLUSION
The paper presents LA as a practical approach to estimating model uncertainty and proposes BO to regularize that uncertainty representation toward the ground truth. It reports better performance than RS and scalability to ImageNet with a large network.
V. CONCLUSION
- The paper combines LA-based model uncertainty estimation with BO to regularize the uncertainty representation toward the ground truth.Source: §V · p. 5
- The framework outperformed RS, demonstrating performance and scalability.Source: §V · p. 5
- The approach scaled to ImageNet using a large deep neural network.Source: §V · p. 5