PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

Fredrik K. Gustafsson, Martin Danelljan, Thomas B. Schön

arXiv:1906.01620v3cs.LGcs.CVstat.ML

TL;DR

Deep neural networks often produce unreliable uncertainty estimates, motivating better scalable methods for epistemic uncertainty in real-world computer vision. The paper proposes a comprehensive evaluation framework and compares ensembling with MC-dropout under challenging synthetic-to-real conditions. It finds that ensembling consistently provides more reliable and practically useful uncertainty estimates.

  • Problem

    DNNs often fail to capture predictive uncertainty, while scalable epistemic-uncertainty methods lacked an extensive real-world comparison needed for safety-critical computer vision.

  • Method

    The paper proposes a framework evaluating scalable methods across regression and classification tasks, using AUSE and calibration metrics, varying M, and testing synthetic-to-real domain shift.

  • Results

    Ensembling consistently provides more reliable and practically useful uncertainty estimates than MC-dropout.

  • Takeaways & Limitations

    The results suggest ensembling should be preferred over MC-dropout for scalable uncertainty estimation in the evaluated real-world computer-vision settings.

  • Takeaways & Limitations

    Semantic-segmentation models trained exclusively on Synscapes have relatively low mIoU because of the intentionally challenging synthetic-to-real domain gap.

Abstract

from arXiv · show

While deep neural networks have become the go-to approach in computer vision, the vast majority of these models fail to properly capture the uncertainty inherent in their predictions. Estimating this predictive uncertainty can be crucial, for example in automotive applications. In Bayesian deep learning, predictive uncertainty is commonly decomposed into the distinct types of aleatoric and epistemic uncertainty. The former can be estimated by letting a neural network output the parameters of a certain probability distribution. Epistemic uncertainty estimation is a more challenging problem, and while different scalable methods recently have emerged, no extensive comparison has been performed in a real-world setting. We therefore accept this task and propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. Our proposed framework is specifically designed to test the robustness required in real-world computer vision applications. We also apply this framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates. Code is available at https://github.com/fregu856/evaluating_bdl.

1. Introduction

The paper addresses unreliable uncertainty estimates in safety-critical computer vision by proposing a comprehensive framework for evaluating scalable epistemic-uncertainty methods. It applies the framework to compare ensembling and MC-dropout under synthetic-to-real domain shift.

  • Motivation: DNNs often fail to capture predictive uncertainty, limiting uncertainty-aware reasoning in safety-critical applications such as automotive vision.Incorrect predictions can therefore remain highly confident, particularly for inputs unlike the training distribution.
  • Motivation: Bayesian deep learning separates predictive uncertainty into epistemic uncertainty from model parameters and aleatoric uncertainty from irreducible data noise.Input-dependent aleatoric uncertainty can arise from noise and ambiguity in the input.
  • Research gap: Existing scalable methods lacked a common, comprehensive evaluation framework, leaving researchers and practitioners unable to properly compare newly proposed approaches.MC-dropout and ensembling were widely used, but previous evaluations were partial, qualitative, fixed in sample count, or restricted in task and metric coverage.
  • Contribution: The comparison reports that ensembling consistently outperforms MC-dropout and provides more reliable and practically useful uncertainty estimates.The framework is presented as an extensive and conclusive comparison of the two scalable methods.
  • Framework: The proposed framework evaluates uncertainty reliability and usefulness across depth completion and semantic segmentation using relative AUSE and absolute calibration metrics.It varies the number of samples M and includes regression and classification tasks.
  • Framework: Models are trained exclusively on synthetic data and evaluated on real-world data to test robustness to the domain shifts found in automotive applications.The framework uses Virtual KITTI and Synscapes for training, with KITTI and Cityscapes examples for real-world validation.

2. Predictive Uncertainty Estimation using Bayesian Deep Learning

The paper explains how DNNs model aleatoric uncertainty by predicting conditional distributions and how approximate Bayesian inference estimates epistemic uncertainty. A toy regression example illustrates why Bayesian inference is needed for reasonable uncertainty outside the training range.

  • DNN prediction: A DNN maps an input x to an output f_θ(x), supporting classification over finite classes and regression over continuous targets.The model is parameterized by θ and can be used to predict a target value y from an input x.
  • Aleatoric Uncertainty: For classification, the DNN predicts logits that Softmax normalizes into a categorical distribution p(y|x, θ), capturing input-dependent aleatoric uncertainty.Maximum-likelihood training uses the categorical cross-entropy objective.
  • Aleatoric Uncertainty: For regression, predicting a Gaussian conditional distribution models input-dependent aleatoric uncertainty through its variance, unlike direct target regression.Direct regression typically minimizes an L1 or L2 loss and does not model aleatoric uncertainty.
  • Epistemic Uncertainty: Bayesian inference estimates epistemic uncertainty by marginalizing model parameters using the posterior p(θ|D), but the required integral is generally intractable.Approximate inference replaces the true posterior with q(θ) and uses M Monte Carlo parameter samples.
  • Epistemic Uncertainty: Approximate Bayesian predictive distributions combine sampled model predictions to estimate both aleatoric and epistemic uncertainty.The approximation quality depends on the number of samples M and the method used to generate q(θ).
  • Illustrative Example: In the toy regression problem, maximum-likelihood Gaussian prediction becomes overly confident outside the training interval, whereas approximate Bayesian inference predicts more reasonable uncertainty there.Training data cover only x ∈ [-3, 3], while the Bayesian model uses HMC with M = 1 000 samples.

3. Related Work

Approximate Bayesian inference uses MCMC or variational methods to construct tractable posterior approximations for deep neural networks. The section also presents MC-dropout and ensembling as scalable alternatives, with ensembling interpretable as approximate Bayesian inference through its implicit sampling distribution.

  • Approximate Bayesian Inference: MCMC approximates the posterior by sampling parameters from a Markov chain whose stationary distribution is p(θ|D), but HMC does not scale to large DNNs or datasets.SG-MCMC replaces full-data gradients with stochastic gradients, although limited computational budgets can hinder exploration of high-dimensional, multimodal posteriors.
  • Approximate Bayesian Inference: Variational inference explicitly chooses a parameterized distribution qφ(θ) and minimizes its KL divergence from the true posterior, but expressive variants require sophisticated implementations.MC-dropout is introduced as a particularly simple and scalable method within this paradigm.
  • Ensembling: Ensembling learns multiple point estimates with random initialization and averages their corresponding parametric predictive models.This construction is presented as a scalable approach to predictive uncertainty estimation.
  • Ensembling: Ensembling can be viewed as approximate Bayesian inference because its learned point estimates act as samples from an implicit distribution ˆq(θ).The approximation depends on how well this implicit distribution matches the posterior, and random initializations can capture multimodality through different local optima.

4. Experiments

The experiments compare ensembling and MC-dropout on toy Bayesian inference problems and real-world depth-completion and semantic-segmentation tasks. Across these evaluations, ensembling generally provides stronger uncertainty estimates, while performance depends on the task and number of samples M.

  • Illustrative Toy Problems: Toy experiments compare ensembling and MC-dropout with SGLD and SGHMC by measuring KL divergence from HMC predictive distributions.The toy regression and classification studies use HMC as a reference distribution and vary the number of samples M.
  • Illustrative Toy Problems: Ensembling consistently outperforms MC-dropout, SGLD, and SGHMC in approximating the HMC predictive distribution.Ensembling provides reasonable HMC approximations even for relatively small M, particularly compared with MC-dropout.
  • Evaluation Framework: The evaluation combines relative uncertainty quality, absolute calibration, and task performance metrics to assess robustness across regression and classification.AUSE measures uncertainty-based ranking, AUCE measures regression calibration, ECE measures classification calibration, and RMSE or mIoU measure predictive performance.
  • Depth Completion: Depth-completion experiments on KITTI compare ensembling and MC-dropout using AUSE, AUCE, and RMSE across different sample counts M.The evaluation uses camera images, sparse LiDAR depth maps, and dense depth targets; MC-dropout uses dropout layers in the network.
  • Depth Completion: Ensembling consistently outperforms MC-dropout on depth-completion AUSE, while its calibration improves with M and MC-dropout progressively worsens.For AUSE, both curves decrease similarly with M; the clearer ranking from calibration shows opposing trends for the two methods, with M = 1 representing aleatoric uncertainty alone.
  • Street-Scene Semantic Segmentation: Semantic-segmentation experiments on Cityscapes compare ensembling and MC-dropout using AUSE, ECE, and mIoU for different M.Both methods improve as M increases, but ensembling generally improves faster and transitions from over-confidence toward slight over-conservatism, whereas MC-dropout remains somewhat over-confident.

5. Discussion & Conclusion

The paper presents a comprehensive robustness-oriented evaluation framework and finds that ensembling provides more reliable and practically useful uncertainty estimates than MC-dropout. It attributes this advantage to ensembling’s ability to capture posterior multimodality, while noting test-time computational costs for both methods.

  • 5. Discussion & Conclusion: Ensembling consistently provides more reliable and practically useful uncertainty estimates than MC-dropout.The authors attribute this success to random initialization capturing important multimodality in the DNN posterior.
  • 5. Discussion & Conclusion: The framework is designed to evaluate scalable epistemic uncertainty estimation under robustness requirements for real-world computer vision.The paper applies it to an extensive comparison of ensembling and MC-dropout.
  • 5. Discussion & Conclusion: MC-dropout has a larger design space, so careful tuning could potentially close the performance gap on individual tasks.The authors identify ensembling’s simplicity and general applicability as key strengths.
  • 5. Discussion & Conclusion: Both methods incur test-time computational costs that grow linearly with the number of samples M, limiting real-time applicability.The authors identify model pruning for predictive uncertainty as a future research direction.

Supplementary Material

The supplementary material documents toy-problem evaluation, predictive-distribution approximations, model setups, optimization procedures, and sampling details for the compared methods.

  • Supplementary Material: The supplementary material contains Appendices A–D with additional experimental and methodological details.It includes further details on the illustrative toy problems experiments.
  • Supplementary Material: For Gaussian regression, the predictive distribution is a uniformly weighted Gaussian mixture approximated by a single Gaussian using its mixture mean and variance.This approximation is described for the predictive distribution in the Gaussian model.
  • Supplementary Material: Toy regression and classification methods are evaluated against HMC-based predictive distributions using KL divergence as the comparison metric.The experiments use a Gaussian regression model and discrete-distribution KL divergence for classification.
  • Supplementary Material: Ensembling and MC-dropout use feed-forward networks with two hidden layers, while dropout is inserted after the first hidden layer for MC-dropout.The dropout probabilities are p = 0.2 for regression and p = 0.1 for classification.
  • Supplementary Material: Ensembling is trained for 150 epochs with Adam, whereas MC-dropout is trained for 300 epochs using the same batch size and learning rate.Both procedures use a batch size of 32 and a fixed learning rate of 0.001.
  • Supplementary Material: SGLD and SGHMC generate samples from parameter trajectories using stochastic-gradient updates with decayed step sizes.The supplementary material specifies trajectory equations, noise terms, step-size schedules, and training durations.

B.3. Description of Results

The supplementary results describe how predictive means and standard deviations were computed for ensembling, MC-dropout, SGLD, and SGHMC across toy-problem comparisons.

  • B.3. Description of Results: The toy-problem results were obtained by computing predictive means and standard deviations from method-specific samples or model sets.The comparison includes ensembling, MC-dropout, SGLD, and SGHMC.
  • B.3. Description of Results: Ensembling used 1024 trained models and formed estimates from 1024/M unique model sets for M ∈ {8, 16, 32, 64, 128, 256}.This procedure varies the number of ensemble members used in each estimate.
  • B.3. Description of Results: MC-dropout estimates were computed from 10 trained models using the same training procedure.The passage does not specify a separate M-dependent grouping procedure for these models.
  • B.3. Description of Results: SGLD estimates were computed from 6 trained models using the same training procedure.The resulting predictive mean and standard deviation were used in the toy-problem comparison.
  • B.3. Description of Results: SGHMC estimates were computed from 6 trained models using the same training procedure.The resulting predictive mean and standard deviation were used in the toy-problem comparison.

B.4. Additional Results

Additional toy-problem comparisons vary the optimizer and training duration. Ensembling remains strongest under the shorter SGLD and SGHMC training budget, although those methods can perform better for regression under longer training.

  • B.4. Additional Results: Ensembling consistently outperforms the compared methods for classification when using SGD or SGD with momentum.The comparison corresponds to the evaluations shown in Figures 10 and 11.
  • B.4. Additional Results: SGLD and SGHMC perform better than ensembling for regression under the SGD and SGD-with-momentum comparisons.This comparison is complicated because SGLD and SGHMC are trained for 256 times longer than each ensemble model.
  • B.4. Additional Results: When SGLD and SGHMC are trained for 64 times longer than each ensemble model, ensembling consistently outperforms them.This shorter relative training comparison is shown in Figure 12.
  • B.4. Additional Results: For Adam-based MC-dropout, the drop probability is varied and the best-performing variant is selected for the reported comparison.The selected variant is marked with an asterisk in Figure 13.

B.5. Qualitative Results

The appendix visualizes predictive distributions produced by ensembling, MC-dropout, SGLD, and SGHMC on toy regression and classification problems. It also documents the depth-completion training setup used for the real-world experiments.

  • Toy Problems: Predictive distributions are visualized for ensembling, MC-dropout, SGLD, and SGHMC.Figures 14–21 cover toy regression and classification examples with M = 64.
  • Toy Problems: Figures 13–21 present illustrative toy problems and quantitative results for the evaluated uncertainty methods.Figure 13 specifically reports MC-dropout using Adam.
  • Depth Completion Setup: Depth-completion models use 40 000 Adam steps, batch size 4, learning rate 10−5, weight decay 0.0005, random 352 × 352 crops, and vertical flips.The smaller batch size, fewer steps, and random crops were chosen to enable repeated experiments.

C.2. Description of Results

The depth-completion results were generated from repeated model training for ensembling and MC-dropout. Ensemble statistics used multiple randomly drawn model sets, while MC-dropout statistics used all trained models.

  • Results Procedure: The described procedure produced the results shown in Figure 6 for the depth-completion experiments in Section 4.2.
  • Ensembling: 33 ensemble models supported mean and standard-deviation estimates across M = 1, 2, 4, 8, 16, and 32.The estimates used 32, 16, or 4 randomly drawn model sets depending on M, without repeating a set.
  • MC-dropout: 16 MC-dropout models were trained, and their mean and standard deviation were computed.

C.3. Additional Results

Additional KITTI depth-completion results compare ensembling and MC-dropout through sparsification plots, sparsification error curves, and calibration plots. The appendix also describes the experimental setup used for these evaluations.

  • Evaluation Visualizations: KITTI depth-completion results include sparsification plots, sparsification error curves, and calibration plots for both methods.Figures 22–27 provide examples and condensed curves or plots for ensembling and MC-dropout.
  • Ensembling: Ensembling results are illustrated with sparsification plots, condensed sparsification error curves, and condensed calibration plots.
  • MC-dropout: MC-dropout results are illustrated with sparsification plots, condensed sparsification error curves, and condensed calibration plots.

D.2. Description of Results

Additional street-scene semantic-segmentation results compare ensembling and MC-dropout using sparsification and reliability visualizations. The appendix also specifies the ensemble training and repeated-model evaluation procedures.

  • Ensembling: Ensembling uses 26 trained models and evaluates M ∈{1, 2, 4, 8, 16} with eight randomly drawn model sets.The same model set was not drawn more than once.
  • MC-dropout: MC-dropout uses eight trained models to compute the mean and standard deviation.
  • Evaluation Visualizations: Semantic-segmentation results include sparsification plots, sparsification error curves, and reliability diagrams for both methods.Figures 28–35 show examples and condensed visualizations for ensembling and MC-dropout.
Loading 1906.01620v3…