PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement

Matias Valdenegro-Toro, Daniel Saromo

arXiv:2204.09308v1cs.LGcs.CV

TL;DR

Reliable predictions require separating aleatoric data uncertainty from epistemic model uncertainty, but existing disentanglement methods were not broadly compared across uncertainty estimators. This paper generalizes the formulation and evaluates it across regression and classification, finding that uncertainty sources interact, Flipout can produce nearly zero epistemic uncertainty, and ensembles generally disentangle best.

  • Problem

    Neural-network applications require confidence estimates, but aleatoric and epistemic uncertainty are often combined rather than separately quantified.

  • Method

    The paper generalizes aleatoric–epistemic disentanglement across uncertainty quantification methods and evaluates the methods in regression and classification.

  • Results

    Ensembles provide the best overall disentangling behavior, while uncertainty sources interact, Flipout produces almost zero epistemic uncertainty, and aleatoric estimates can be unreliable out of distribution.

  • Takeaways & Limitations

    The findings support choosing ensembles for disentangled uncertainty and motivate further research on interactions between aleatoric and epistemic estimates.

  • Takeaways & Limitations

    The study identifies unresolved interactions between aleatoric and epistemic uncertainty and the need for methods that reduce interactions affecting aleatoric uncertainty.

Abstract

from arXiv · show

Neural networks are ubiquitous in many tasks, but trusting their predictions is an open issue. Uncertainty quantification is required for many applications, and disentangled aleatoric and epistemic uncertainties are best. In this paper, we generalize methods to produce disentangled uncertainties to work with different uncertainty quantification methods, and evaluate their capability to produce disentangled uncertainties. Our results show that: there is an interaction between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on aleatoric uncertainty, some methods like Flipout produce zero epistemic uncertainty, aleatoric uncertainty is unreliable in the out-of-distribution setting, and Ensembles provide overall the best disentangling quality. We also explore the error produced by the number of samples hyper-parameter in the sampling softmax function, recommending N > 100 samples. We expect that our formulation and results help practitioners and researchers choose uncertainty methods and expand the use of disentangled uncertainties, as well as motivate additional research into this topic.

1. Introduction

The paper motivates disentangling aleatoric data uncertainty from epistemic model uncertainty and generalizes this analysis across uncertainty-estimation methods. Experiments compare methods across regression and classification, finding interactions between uncertainty sources, anomalous Flipout behavior, unreliable out-of-distribution aleatoric estimates, and strong ensemble performance.

  • Motivation: Aleatoric uncertainty reflects data noise, whereas epistemic uncertainty reflects model uncertainty and can be useful for active learning and out-of-distribution detection.Predictive uncertainty commonly combines both components, but applications may require treating them differently.
  • Approach: The paper generalizes Kendall and Gal's disentanglement formulation beyond MC-Dropout to multiple uncertainty quantification methods.The comparison includes dropout, dropconnect, ensembles, and Flipout.
  • Evaluation: Experiments evaluate disentangling quality across regression and classification tasks, including classification on FER+.The study also examines interactions between aleatoric and epistemic uncertainty.
  • Findings: Aleatoric and epistemic uncertainty interact, contrary to the expectation that aleatoric uncertainty should not depend on the model.This interaction is reported as an unexpected finding.
  • Findings: Flipout can output only aleatoric uncertainty and zero epistemic uncertainty, while aleatoric estimates are unreliable out of distribution and ensembles show the best disentangling behavior.The out-of-distribution unreliability is particularly reported for regression.
  • Findings: The study recommends N = 100 sampling-softmax samples to prevent classification errors caused by approximation error.The sample-count recommendation addresses the sampling-softmax hyper-parameter.

2. Related Work

Prior work separates total uncertainty into aleatoric and epistemic components using model-specific approaches. The paper situates its method among regression, reinforcement-learning, and stochastic neural-network sampling techniques, including sampling softmax for uncertain logits.

  • Disentanglement methods: Existing approaches estimate total and aleatoric uncertainty and derive epistemic uncertainty, but some have been tested only in regression and reinforcement learning.This motivates broader evaluation across uncertainty-estimation settings.
  • Sampling methods: Monte Carlo Dropout estimates prediction uncertainty by turning off activations during repeated forward passes, while Monte Carlo DropConnect turns off weights.These are described as common sampling methods for stochastic prediction.
  • Sampling softmax: Sampling softmax visualizes binary probability distributions generated from Gaussian logit distributions with different means and standard deviations.The figure compares how logit-distribution parameters affect the resulting probabilities.
  • Sampling softmax: When logit means differ, logit variances have a more significant effect on the final probabilities; equal means produce less intuitive behavior.The paper uses these visualizations to motivate examination of the sampling-softmax function.

3. Uncertainty Disentanglement

The paper generalizes uncertainty disentanglement across sampling and ensemble methods, combining forward-pass predictions into aleatoric and epistemic components. It also analyzes classification sampling-softmax behavior and the sample count needed for reliable probabilities.

  • Predictive uncertainty: Uncertainty methods generate multiple means and variances that are combined into a predictive Gaussian mixture.The predictive mean averages sample means, while predictive variance combines averaged variances with mean variability.
  • Uncertainty decomposition: Aleatoric uncertainty is the mean of sampled variances, whereas epistemic uncertainty is the variance of sampled means.This decomposition follows the law of total variance.
  • Regression: The Gaussian negative log-likelihood trains variance heads, but it can underestimate variance; β-NLL modifies it by weighting examples according to predicted variance.A stop-gradient operation prevents gradients through the variance weighting, while β controls weighting strength.
  • Classification: For classification, aleatoric and epistemic logit variances are separately passed through sampling softmax and entropy to obtain scalar uncertainty measures.The resulting probabilities or entropies do not generally sum to predictive probabilities or entropy; only logits can be summed for predictive logits.
  • Sampling softmax: Sampling softmax averages softmax outputs from Gaussian logits whose mean and variance are predicted by two fully connected layers.Logit variances affect final probabilities more strongly when the logit means differ.
  • Sample-count tuning: At least N = 100 samples are required to obtain zero classification error from sampling error in the evaluated settings.The study compares variable N against an N = 100000 approximation using L2 probability error and misclassification probability.

4. Experimental Comparison

The experiments compare uncertainty methods across toy regression and FER+ classification, showing that disentangling quality depends on both the method and loss. Ensembles generally provide the strongest overall behavior, while Flipout often produces little or no epistemic uncertainty.

  • Uncertainty Methods: The study compares MC-Dropout, MC-DropConnect, Ensembles, and Flipout for disentangling uncertainty across regression and classification tasks.Evaluation uses M = 20 forward passes, with N = 100 sampling-softmax samples for classification.
  • Toy Regression: Dropout and DropConnect overestimate epistemic uncertainty on the training set, while Ensembles and Flipout show low uncertainty outside it under the NLL loss.These comparisons use a toy sinusoid problem with homoscedastic and heteroscedastic aleatoric uncertainty.
  • Toy Regression: β-NLL increases Ensembles’ epistemic uncertainty outside the training set, while Dropout and DropConnect still overestimate epistemic uncertainty on the training set.The β-NLL comparison evaluates the same toy sinusoid setting with homoscedastic and heteroscedastic aleatoric uncertainty.
  • Toy Regression: Aleatoric uncertainty is unreliable out of distribution, with Ensembles and Flipout producing constant variances instead of learning the increasing noise pattern.In the experiment, out-of-distribution inputs are x > 10; the Classical NN learned increasing aleatoric uncertainty there.
  • Discussion: Disentanglement quality depends on both the uncertainty quantification model and the NLL loss choice, affecting aleatoric and epistemic estimates.This interaction is expected for epistemic uncertainty but not for aleatoric uncertainty.
  • Overall Comparison: Flipout produces almost zero epistemic uncertainty in both classification and regression, making it unsuitable when uncertainty disentanglement is required.The paper describes this behavior as unexpected and most likely incorrect.
  • Loss Effects: β-NLL improves aleatoric uncertainty and often epistemic uncertainty for Dropout and DropConnect, but Flipout’s epistemic uncertainty decreases with β-NLL.For Ensembles, β-NLL produces increasing epistemic uncertainty in the output-distribution setting, whereas NLL produces smaller epistemic uncertainty.
  • Overall Comparison: Ensembles achieve the best overall combination for disentanglement, using β-NLL for regression and Ensembles alone for classification.The paper reports best FER+ accuracy for Ensembles and good disentangling in the toy regression experiments.

5. Conclusions and Future Work

The paper generalizes disentangled uncertainty methods, compares their behavior, and identifies interactions and approximation requirements that shape practical use.

  • The study generalizes aleatoric–epistemic uncertainty disentanglement and compares uncertainty quantification methods across regression and classification.
  • Ensembles show the best disentangling quality, using β-NLL for regression and the sampling softmax function for classification.
  • At least N = 100 sampling-softmax samples are required for a good approximation and near-zero misclassification error.
  • The results can guide method selection and motivate disentangling quality as an additional metric for future uncertainty methods.
  • Further research is needed to understand aleatoric–epistemic interactions and reduce interactions affecting aleatoric uncertainty.

A. Training Data for Toy Regression Example

The toy regression evaluation uses training data with input-dependent, heteroscedastic aleatoric uncertainty that increases linearly with x.

  • Aleatoric uncertainty increases linearly with x in the toy regression training data.
  • The toy sinusoid problem uses training data with heteroscedastic aleatoric uncertainty.

B. Neural Network Architecture Details

The evaluation uses separate regression and FER+ classification tasks with method-specific network configurations, including baseline, stochastic, Flipout, and ensemble models.

  • Tasks: The experiments comprise a sinusoidal regression task and a FER+ classification task.
  • Regression configuration: The regression networks use 700 epochs, batch size 32, and Adam with learning rate 0.001, β1 = 0.9, and β2 = 0.999.
  • Regression configuration: Regression methods include a baseline, Dropout, DropConnect, Flipout, and Ensembles of five baseline copies with different random initializations.
  • Uncertainty architecture: For regression uncertainty, parallel heads predict the mean μ(x) and standard deviation σ(x), as represented in the network architecture diagram.
  • Classification configuration: The classification networks use 120 epochs, batch size 64, and Adam with learning rate 0.001, β1 = 0.9, and β2 = 0.999.
  • Classification configuration: Classification methods include baseline, Dropout, DropConnect, Flipout, and Ensembles, with convolutional architectures and method-specific stochastic layers.
Loading 2204.09308v1…