Source-linked AI summary
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Alex Kendall, Yarin Gal
TL;DR
Modeling uncertainty in computer vision requires distinguishing observation noise from uncertainty in the model. This paper unifies input-dependent aleatoric and epistemic uncertainty for regression and classification, achieving state-of-the-art results on depth regression and semantic segmentation benchmarks.
Problem
Epistemic uncertainty has been difficult to model in computer vision, limiting clear assessment of observation noise versus model uncertainty.
Method
The paper presents a Bayesian deep learning framework combining input-dependent aleatoric uncertainty with epistemic uncertainty for regression and classification.
Results
The combined uncertainty models achieve new state-of-the-art results on depth regression and semantic segmentation benchmarks, improving performance 1 −3% over non-Bayesian baselines.
Takeaways & Limitations
Modeling aleatoric and epistemic uncertainty together supports robust vision predictions across noisy-data attenuation, depth regression, and semantic segmentation.
Takeaways & Limitations
Real-time epistemic uncertainty remains an important open direction because Monte Carlo dropout can cause substantial computational slowdowns.
Abstract
from arXiv · showhide
There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model -- uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.
1 Introduction
The section distinguishes aleatoric uncertainty, arising from irreducible observation noise, from epistemic uncertainty, arising from model ignorance and reducible with more data. It introduces a unified Bayesian deep learning framework for vision tasks and reports improved performance from explicitly modeling aleatoric uncertainty.
- Motivation: Deep learning vision systems can make inaccurate predictions when their outputs are accepted without quantifying what the model does not know.The introduction motivates uncertainty estimation with consequential failures of perception systems, including a fatality involving assisted driving.
- Types of uncertainty: Aleatoric uncertainty captures irreducible observation noise, whereas epistemic uncertainty captures model uncertainty that can be explained away with sufficient data.Aleatoric uncertainty includes sensor or motion noise; aleatoric uncertainty may be heteroscedastic when noise depends on the input.
- Framework: The paper presents a unified Bayesian deep learning framework that combines learned input-dependent aleatoric uncertainty with approximations of epistemic uncertainty for depth regression and semantic segmentation.The framework covers regression and classification, including per-pixel depth regression and semantic segmentation.
- Contributions: 1 −3% improvement over non-Bayesian baselines is achieved by reducing the effect of noisy data through explicit aleatoric uncertainty and its implied attenuation.The paper also studies the trade-offs between aleatoric and epistemic uncertainty, including their properties, model performance, and inference time.
2 Related Work
Related work distinguishes epistemic uncertainty over model parameters from aleatoric uncertainty in observations, with Bayesian neural networks and approximate inference providing practical tools for modeling the former. Heteroscedastic regression models input-dependent observation noise but do not capture epistemic uncertainty, motivating a unified treatment.
- Uncertainty types: Existing Bayesian deep-learning approaches typically model either epistemic uncertainty over weights or aleatoric uncertainty over model outputs.Epistemic uncertainty is represented through a distribution over model parameters, whereas aleatoric uncertainty is represented through a distribution over observations or outputs.
- Bayesian neural networks: Bayesian neural networks replace deterministic weights with distributions and use marginalisation over possible weights, but exact inference is analytically intractable.Approximate inference fits a tractable distribution to the posterior over weights instead of averaging over all weights directly.
- Approximate inference: Dropout variational inference approximates Bayesian inference by applying dropout during training and test-time stochastic forward passes.The approximation seeks a tractable distribution minimizing KL divergence to the true posterior and can be interpreted as a mixture of two Gaussians.
- Aleatoric uncertainty: Heteroscedastic regression allows observation noise to vary with input, unlike homoscedastic regression, which assumes constant noise across inputs.Input-dependent noise is useful when different parts of the observation space exhibit different noise levels.
- Limitations and motivation: MAP-based heteroscedastic models estimate a single parameter value and therefore capture data-dependent noise without modeling epistemic uncertainty.The paper next combines both uncertainty types and interprets heteroscedastic noise as model attenuation.
3 Combining Aleatoric and Epistemic Uncertainty in One Model
The section develops a Bayesian vision model that combines epistemic uncertainty over weights with input-dependent aleatoric uncertainty. It extends learned loss attenuation from regression to heteroscedastic classification through uncertainty in logit space.
- Combined Bayesian model: The proposed model converts a heteroscedastic neural network into a Bayesian neural network by placing a distribution over its weights.This construction is developed specifically for vision models and captures both epistemic and aleatoric uncertainty.
- Combined Bayesian model: Dropout variational inference approximates the posterior over weights, while stochastic weight samples produce predictive means and variances.A single Bayesian convolutional network uses a split head to predict both the output ˆy and variance ˆσ2.
- Learned loss attenuation: The regression loss learns aleatoric variance without uncertainty labels by combining stochastic residual regression with an uncertainty regularization term.The regularizer penalizes excessive uncertainty, while low predicted variance increases penalties for large residuals.
- Learned loss attenuation: Predicted uncertainty attenuates residual weighting by exp(−si), reducing the influence of noisy or erroneously labeled inputs.This gives the model an intelligent robust regression behavior and makes it more robust to noisy data.
- Heteroscedastic classification: The classification extension places heteroscedastic regression uncertainty over logits and marginalizes it before applying the softmax.Because the expected log likelihood lacks an analytic solution, the objective is approximated with fast Monte Carlo sampling through the softmax.
4 Experiments
Experiments on semantic segmentation and monocular depth regression show that jointly modeling aleatoric and epistemic uncertainty improves performance across multiple datasets. The results also indicate that aleatoric uncertainty captures inherent task difficulty, while epistemic uncertainty reflects limited training data.
- Experimental setup: The framework is evaluated on pixel-wise depth regression and semantic segmentation across CamVid, Make3D, and NYUv2 datasets.The experiments are designed to test the robustness of learned loss attenuation from uncertainty modeling.
- Experimental setup: The models combine Monte Carlo dropout for epistemic uncertainty with aleatoric loss functions for regression and classification.The DenseNet-based models use dropout probability p = 0.2 after each convolutional layer and 50 Monte Carlo samples.
- Semantic segmentation: Modeling both aleatoric and epistemic uncertainty improves semantic segmentation accuracy over state-of-the-art baselines, including DeepLabLargeFOV on NYUv2.NYUv2 is more challenging than CamVid because it has less scene structure and 40 semantic classes.
- Depth regression: On monocular depth regression, combining uncertainties improves accuracy and captures distinct failure sources in the task.Aleatoric uncertainty is higher for large depths, reflective surfaces, and occlusion boundaries, whereas epistemic uncertainty is higher for rare objects such as humans.
- Semantic segmentation: 67.5% mean intersection over union is achieved on CamVid, where modeling both uncertainties outperforms the baseline and their combination performs best.Aleatoric loss attenuation provides a larger improvement than epistemic uncertainty alone on this application.
5 Analysis: What Do Aleatoric and Epistemic Uncertainties Capture?
The analysis finds that aleatoric uncertainty reflects irreducible observation noise, while epistemic uncertainty tracks model unfamiliarity and can diminish with more data. Uncertainty estimates correlate with prediction accuracy, but epistemic modeling incurs substantial computational costs.
- Uncertainty quality: All precision-recall curves are strictly decreasing, showing that the uncertainty measurements correlate well with prediction accuracy.The curves are obtained by removing pixels whose uncertainty exceeds varying percentile thresholds.
- What uncertainties capture: Aleatoric uncertainty remains relatively constant as the training dataset grows, indicating that additional data cannot explain it away.This behavior is contrasted with epistemic uncertainty, which decreases as the training dataset gets larger.
- What uncertainties capture: Epistemic uncertainty increases considerably on test points far from the training distribution, whereas aleatoric uncertainty does not increase for out-of-data examples.The result supports using epistemic uncertainty to capture situations not encountered during training.
- Safety implications: Epistemic uncertainty is required to detect previously unseen situations in safety-critical systems.The analysis links this requirement to epistemic uncertainty’s increase on examples far from the training data.
- Computational cost: A DenseNet model processes a 640×480 image in 150ms on an NVIDIA Titan X GPU, while aleatoric uncertainty adds negligible compute.Epistemic uncertainty requires expensive Monte Carlo dropout sampling, and sampling the entire DenseNet is difficult to parallelize because of GPU memory constraints.
6 Conclusions
The paper concludes that aleatoric and epistemic uncertainty serve complementary purposes across data regimes and applications, while their combination achieves state-of-the-art vision results. It also identifies real-time epistemic uncertainty as an important direction for future work.
- Conclusions: The framework combines input-dependent aleatoric uncertainty with epistemic uncertainty for both regression and classification.Aleatoric uncertainty is learned from input data on top of epistemic uncertainty models.
- Conclusions: Aleatoric uncertainty is important for large datasets, where epistemic uncertainty is explained away, and real-time applications without expensive Monte Carlo samples.The framework forms aleatoric models without requiring expensive Monte Carlo sampling.
- Conclusions: Epistemic uncertainty is important for safety-critical applications involving examples different from training data and for small datasets with sparse training data.These settings require modeling uncertainty in the model rather than only observation noise.
- Conclusions: Combining aleatoric and epistemic uncertainty achieves new state-of-the-art results on depth regression and semantic segmentation benchmarks.The two uncertainty models are complementary rather than mutually exclusive.
- Conclusions: Real-time epistemic uncertainty in deep learning remains an important direction for future research.The paper motivates this direction through disasters that could have been averted by real-time Bayesian deep learning tools.