PaperlayerView exact revision on arXiv

Source-linked AI summary

Getting a CLUE: A Method for Explaining Uncertainty Estimates

Javier Antorán, Umang Bhatt, Tameem Adel, Adrian Weller, José Miguel Hernández-Lobato

arXiv:2006.06848v2stat.MLcs.LG

TL;DR

Predictive uncertainty is important for trustworthy machine learning, but relatively little research has explained what leads to neural networks' uncertainty. CLUE finds small, in-distribution input changes that reduce uncertainty; the paper reports that it outperforms simple baselines and that users better predict model behavior after seeing CLUEs.

  • Problem

    There has been relatively little research on explaining neural networks' predictive uncertainty, although uncertainty estimates matter for trustworthy decisions.

    Source: §1 · p. 1

  • Method

    CLUE searches for small input changes in a deep generative model's latent space to reduce uncertainty while keeping explanations in distribution.

    Source: §1 · p. 2

  • Results

    CLUE outperforms simple baselines quantitatively, and users better predict model behavior after being exposed to CLUEs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Users exposed to CLUEs are better able to predict their models' behavior.

    Source: §6 · p. 9

    CLUE does not impose constraints encouraging highlighted pixels to be contiguous, though the authors note these might be useful for natural images.

    Source: §3 · p. 5

Abstract

from arXiv · show

Both uncertainty estimation and interpretability are important factors for trustworthy machine learning systems. However, there is little work at the intersection of these two areas. We address this gap by proposing a novel method for interpreting uncertainty estimates from differentiable probabilistic models, like Bayesian Neural Networks (BNNs). Our method, Counterfactual Latent Uncertainty Explanations (CLUE), indicates how to change an input, while keeping it on the data manifold, such that a BNN becomes more confident about the input's prediction. We validate CLUE through 1) a novel framework for evaluating counterfactual explanations of uncertainty, 2) a series of ablation experiments, and 3) a user study. Our experiments show that CLUE outperforms baselines and enables practitioners to better understand which input patterns are responsible for predictive uncertainty.

1 INTRODUCTION

Predictive uncertainty matters for trustworthy decisions, but practitioners have had few ways to understand which input patterns produce it. CLUE explains uncertainty through small, in-distribution input changes and complements prediction explanations when a probabilistic model is uncertain.

1 INTRODUCTION

  • Predictive uncertainty can help practitioners identify underrepresented groups in training data and direct attention to cases where a model's output should not be relied upon.For example, the introduction describes collecting more data from underrepresented groups and having a doctor direct attention appropriately when a system is uncertain.Source: §1 · p. 1
  • CLUE addresses limited research on explaining neural-network predictive uncertainty by identifying input-space features responsible for uncertainty in probabilistic models, particularly BNNs.The authors describe CLUE as, to their knowledge, the first approach to shed light on these features.Source: §1 · p. 1
  • When the BNN is uncertain, CLUE explains the uncertainty; when it is confident, existing feature-importance methods can explain the decision.The authors note that uncertain predictions may be wrong and their explanations disproportionately affected by factors unrelated to input patterns.Source: Figure 1 · p. 2, §1 · p. 2
  • CLUE seeks the smallest change to an input that keeps it in distribution while making the model more certain about its decision.The method is described as applicable to tabular and image data, and to classification and regression.Source: §1 · p. 2
  • The paper contributes a computational evaluation framework, baseline comparisons, ablations, and a user study of whether practitioners can predict BNN uncertainty.The study reports that CLUE helps practitioners predict whether new inputs will receive high or low uncertainty.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries distinguish sources of predictive uncertainty, explain why sensitivity-based explanations can be unhelpful, and introduce counterfactual explanations as a way to describe input changes. CLUE builds on these ideas by searching a generative model's latent space to keep explanations in distribution.

2.1 UNCERTAINTY IN BNNS

  • Aleatoric uncertainty reflects noise in the data, while epistemic uncertainty reflects limited knowledge about model parameters and can arise away from the training manifold.For BNNs, parameter uncertainty is translated into predictive uncertainty through marginalization; Figure 2 illustrates the two forms on Moons.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 UNCERTAINTY SENSITIVITY ANALYSIS

  • Sensitivity explanations use linear approximations, but nonlinear uncertainty and high-dimensional inputs can make their explanations misleading or point away from the data manifold.CLUE instead searches a DGM's latent space to avoid crude linear approximations and keep explanations in distribution.Source: §2.2 · p. 3

2.3 COUNTERFACTUAL EXPLANATIONS

  • Here, counterfactual explanations seek small input changes that reduce uncertainty without explicit causal assumptions; searching an auxiliary DGM's latent space constrains them to the data manifold.For an out-of-distribution input, CLUE returns the nearest in-distribution analogue.Source: §2.3 · p. 4

3 PROPOSED METHOD

CLUE searches a deep generative model’s latent space for inputs that remain similar to an observation but receive lower uncertainty estimates. It decodes the optimized latent code into a counterfactual input.

3 PROPOSED METHOD

  • CLUE uses a VAE decoder as its generative model, with the encoder’s latent mean providing the initial code.The paper denotes the encoder and decoder predictive means as µφ(z|x) and µθ(x|z), respectively.Source: §3 · p. 4
  • CLUE minimizes an objective combining predictive uncertainty and distance from the original input, then decodes the minimizing latent code into a counterfactual.The procedure backpropagates uncertainty gradients through the decoder into latent space.Source: §3 · p. 4, Figure 4 · p. 5
  • The distance term balances input and prediction similarity; CLUE uses L1 input distance for sparse explanations, with task-specific prediction distances.Prediction distance is mean squared error for regression and cross-entropy for classification.Source: §3 · p. 4
  • The main experiments set λy to 0 and optimize with Adam, starting from the encoder’s latent mean and stopping after convergence or 35 iterations.λx is selected by cross-validation; optimization runs for at least three iterations, with early stopping after three consecutive small decreases in loss.Source: §3 · p. 4
  • Repeated optimization from perturbed latent initializations produces diverse CLUEs; σ0 = 0.15 provides a reported trade-off between optimization speed and diversity.The authors use non-convexity of the objective to obtain different CLUEs for the same input.Source: §3 · p. 5
  • Visualizations encode tabular changes by percentile shifts and image changes by a sign-preserving quadratic pixel difference; pixel-contiguity constraints are not used.Continuous tabular features are highlighted at shifts of at least 15 percentile points, while all discrete-variable changes are highlighted; contiguity constraints may be useful for natural images.Source: §3 · p. 5, Figure 5 · p. 5

4 A FRAMEWORK FOR EVALUATING COUNTERFACTUAL EXPLANATIONS OF UNCERTAINTY COMPUTATIONALLY

The authors propose a computational framework for evaluating whether uncertainty counterfactuals are informative and relevant. It uses a conditional generative model to provide synthetic data and ground-truth measures for those evaluations.

4 A FRAMEWORK FOR EVALUATING COUNTERFACTUAL EXPLANATIONS OF UNCERTAINTY COMPUTATIONALLY

  • The framework evaluates counterfactuals for informativeness and relevance, criteria that require access to the data-generating process.Informativeness concerns features affecting BNN uncertainty; relevance requires proximity to the original input and the data manifold.Source: §4 · p. 5
  • A ground-truth VAEAC models inputs and targets, enabling evaluation of counterfactual likelihood and ground-truth target uncertainty.The conditional target distribution supplies ground-truth uncertainty, while log density evaluates whether counterfactuals are in-distribution.Source: §4 · pp. 5–6
  • The evaluation pipeline trains the ground-truth VAEAC, samples synthetic data to train a BNN and auxiliary DGM, generates counterfactuals, then evaluates them with ground-truth conditionals and likelihood.Figure 6 distinguishes the ground-truth VAEAC from an auxiliary DGM, which is absent for methods that do not use one.Source: Figure 6 · p. 6, §4 · p. 6
  • Informativeness is measured by ground-truth aleatoric-uncertainty reduction and, for epistemic uncertainty, by reduced BNN prediction error; relevance uses likelihood and L1 distance.The epistemic measure compares prediction errors before and after the counterfactual; the relevance measures are ground-truth likelihood and distance from the original input.Source: §4 · p. 6

5 EXPERIMENTS

The experiments evaluate CLUE against counterfactual baselines using computational measures and a practitioner-facing user study, then examine how its auxiliary generative model affects explanations. CLUE explains away more uncertainty than U-FIDO, performs best in 8/10 data-manifold tasks, and supports more accurate user predictions than the tested alternatives.

5 EXPERIMENTS

  • CLUE was tested on roughly the 20% most uncertain test points across five datasets, against localized Uncertainty Sensitivity and U-FIDO baselines.The datasets include LSAT, Wine, Credit, COMPAS, and MNIST.Source: §5 · p. 6

5.1 COMPUTATIONAL EVALUATION

  • CLUE explains away more uncertainty than U-FIDO, while U-FIDO always produces smaller input changes.The comparison uses informativeness and input-space distance as competing criteria.Source: Figure 7 · p. 7, §5.1 · p. 7
  • On artificial data for which counterfactuals are generated, CLUE performs best in 8/10 tasks, and generating counterfactuals with a VAE ensures CLUE’s relevance; analogous real-data experiments yield similar results.Source: §5.1 · p. 7

5.2 USER STUDY

  • The user study asked participants to predict whether a test point’s BNN uncertainty exceeded a threshold, using certain and uncertain context points.It compared randomly selected, Local Sensitivity, CLUE, and human-selected certain context points; ten ML graduate students took each survey variant.Source: §5.2 · p. 8
  • 82.22% (CLUE) versus 62.22% (Human CLUE), 61.67% (Random), and 52.78% (Local Sensitivity) was the mean participant accuracy.The study reports p-values of 2.34e−5, 1.47e−5, and 2.60e−9 for CLUE versus Human CLUE, Random, and Sensitivity, respectively.Source: §5.2 · p. 8

5.3 ANALYSIS OF CLUE’S AUXILIARY DEEP GENERATIVE MODEL

  • On high-dimensional MNIST, encoder-mean initialization produces more relevant CLUEs than zero initialization; results are indistinguishable on other datasets.The authors report similar behavior on Credit, their second-highest-dimensional dataset, and note that non-encoder DGMs need an appropriate initialization strategy.Source: §5.3 · p. 9
  • More expressive auxiliary DGMs produce sparser, more relevant CLUEs while preserving the counterfactuals’ predictive entropy.Low-capacity VAEs can introduce unnecessary changes because they fail to reconstruct features that affect the BNN’s uncertainty; the authors note that unlabeled data can support training more powerful DGMs.Source: §5.3 · p. 9

6 CONCLUSION

CLUE explains which input features can be changed to reduce a probabilistic model’s uncertainty, and the paper reports that it outperforms simple baselines and helps users predict model behavior. The appendix describes the datasets, BNN inference and uncertainty estimates, and additional modeling details.

6 CONCLUSION

  • CLUE reveals input features that can be changed to reduce uncertainty in probabilistic models, and users better predicted model behavior after seeing CLUEs.The paper also reports a computational evaluation framework for counterfactual uncertainty explanations.Source: §6 · p. 9

A DATASETS

  • The experiments use five publicly available datasets: four tabular datasets and one image dataset.The dataset table identifies the COMPAS variant as having seven features.Source: A DATASETS · p. 15, Table 3 · p. 15

B.1 INFERENCE IN BNNS

  • The BNNs use Monte Carlo inference with scale-adapted SG-HMC, which estimates the mass matrix and gradient-noise scale during burn-in.The paper specifies a fixed step size of 0.01 and batch size of 512.Source: §B.1 · p. 16

B.2 COMPUTING UNCERTAINTY ESTIMATES

  • For classification, the authors approximate the predictive posterior with Monte Carlo samples and quantify predictive uncertainty using entropy, split into aleatoric and epistemic components.The epistemic component is the difference between total and aleatoric entropy and is also identified as mutual information between the target and weights.Source: §B.2 · p. 16
  • For regression, the BNN uses heteroscedastic Gaussian likelihoods; the variance of the resulting Gaussian mixture serves as the uncertainty metric.The variance decomposes into aleatoric and epistemic components, estimated with Monte Carlo.Source: §B.2 · pp. 16–17
  • On the moons dataset, aleatoric entropy is near zero, while epistemic entropy grows vertically away from the data.The authors relate this pattern to little noise in the dataset, neural-network inductive biases, and the difficulty of characterizing the BNN weight posterior.Source: §B.2 · p. 17, Figure 11 · p. 17

B.3 ARCHITECTURES AND OTHER NETWORK HYPERPARAMETERS

The appendix documents the networks, input representations, and hyperparameter choices used for the experiments. It describes residual architectures, data-type-specific likelihoods, normalization, and dimensionality-adjusted similarity weights.

B.3 ARCHITECTURES AND OTHER NETWORK HYPERPARAMETERS

  • BNNs and tabular-data auxiliary VAEs use fully connected networks with residual connections; MNIST encoders and decoders use six convolutional bottleneck residual blocks.The ground-truth VAEACs use the same architecture as the corresponding auxiliary models, with a slightly larger joint input space.Source: §B.3 · p. 17
  • Ground-truth VAEACs use a two-stage VAE configuration, with lower-level VAEs using two hidden layers.The lower-level models use 300 hidden units for MNIST and 150 for other datasets.Source: §B.3 · p. 17
  • The generative models use RAdam, while input likelihoods vary by data type: Gaussian for continuous features, categorical for discrete features, and Bernoulli for MNIST pixels.For CLUE optimization, gradients through one-hot vectors use the biased softmax straight-through estimator.Source: §B.3 · pp. 17–18

B.4 CLUE HYPERPARAMETERS

  • Continuous features are normalized to zero mean and unit variance, and the CLUE similarity weight is scaled by input dimensionality.The appendix notes a larger scaled weight for MNIST because many pixels are always black; dataset-specific weights and rejection thresholds are tabulated.Source: §B.3 · p. 18, §B.4 · p. 18, Table 5 · p. 18

C MULTIPLICITY OF CLUES

The appendix illustrates that repeated CLUE optimization can yield multiple counterfactuals for one uncertain input, while some initializations may fail to reduce uncertainty substantially. Examples span MNIST digits and COMPAS cases.

C MULTIPLICITY OF CLUES

  • Different latent initializations can yield CLUEs for different predicted MNIST classes that still resemble the original digit.The authors suggest that this multiplicity could inform practitioners about an input’s similarities to multiple classes.Source: C MULTIPLICITY OF CLUES · p. 18
  • Some CLUE runs fail to produce a feature configuration with substantially lower uncertainty, which the authors attribute to disadvantageous latent initialization.Different initializations can explain away different amounts of uncertainty.Source: C MULTIPLICITY OF CLUES · p. 19
  • For one COMPAS individual, changing prior convictions and charge degree, sex and age range, or both sets can reduce uncertainty.Figure 13 shows four candidate CLUEs, of which the first three pass the rejection threshold.Source: C MULTIPLICITY OF CLUES · p. 20, Figure 13 · p. 20

D SENSITIVITY ANALYSIS IN HIGH DIMENSIONAL SPACES

Direct input-space entropy gradients can produce off-manifold, noisy uncertainty explanations in high dimensions. CLUE instead visualizes uncertainty-reducing changes along a latent-space trajectory.

D SENSITIVITY ANALYSIS IN HIGH DIMENSIONAL SPACES

  • In high-dimensional inputs, −∇xH can leave the data manifold, producing noisy, potentially meaningless explanations even when entropy falls.Aggregating such gradient steps across test points can yield a white-noise-like sensitivity map.Source: D SENSITIVITY ANALYSIS IN HIGH… · p. 20, Figure 14 · p. 20

E VISUALIZING OPTIMIZATION IN LATENT SPACE

F COMPARING CLUE TO FEATURE IMPORTANCE ESTIMATORS

CLUE explains uncertainty through changes to inputs, unlike feature-importance methods that attribute evidence for a selected class. It is class-agnostic and can suggest adding absent features; examples show it often reduces uncertainty more than U-FIDO.

F COMPARING CLUE TO FEATURE IMPORTANCE ESTIMATORS

  • The examples compare LIME and Kernel SHAP on both high-confidence and above-threshold-uncertainty MNIST digits, using the top predicted classes.The MNIST LIME setup uses default hyperparameters and plots its top 10 segments with weight greater than 0.01.Source: Figure 16 · p. 21, F COMPARING CLUE TO FEATURE IMPORTANCE… · p. 22, Figure 17 · p. 22
  • CLUE attributions identify regions to add or remove to increase certainty, whereas feature-importance attributions indicate evidence for a predicted class.Positive CLUE attribution marks a feature whose addition increases certainty; negative attribution marks one whose absence increases certainty.Source: F COMPARING CLUE TO FEATURE IMPORTANCE… · pp. 22–23
  • Feature-importance methods cannot add features and are limited to existing inputs, which may produce noisy explanations when inputs omit information needed for a prediction.CLUE can suggest adding unobserved features, a capability the paper identifies as unique.Source: F COMPARING CLUE TO FEATURE IMPORTANCE… · p. 23
  • Unlike LIME and SHAP, CLUE is class-agnostic, avoiding the need to choose a class when several classes have similarly high probability.LIME locally fits a surrogate linear model; Kernel SHAP assigns importance relative to a reference input.Source: F COMPARING CLUE TO FEATURE IMPORTANCE… · pp. 21, 23

G ADDITIONAL CLUE AND U-FIDO EXAMPLES

H.1 ABLATION EXPERIMENTS

The ablations show that CLUE depends on suitable initialization, generative-model capacity, and output-space regularization. Real-data checks broadly match the computational evaluation, while the additional user-study analysis finds CLUE strongest on certain test points.

H.1 ABLATION EXPERIMENTS

  • Encoder-based initialization and zero initialization are indistinguishable on LSAT, COMPAS, and Wine, while encoder initialization keeps Credit CLUEs slightly closer to inputs.The difference is largest on MNIST; the authors conjecture encoder initialization can leave optimization in a local basin, sometimes changing only features with high uncertainty-reduction potential.Source: §H.1 · p. 25, Figure 20 · p. 25
  • Higher-capacity VAEs preserve more input uncertainty during autoencoding, allowing CLUE to explain more away; low-capacity models can yield unchanged inputs or large shifts with little uncertainty reduction.The required latent-space size depends on dataset dimensionality.Source: §H.1 · p. 26, Figure 21 · p. 26
  • Increasing λy reduces class changes between inputs and CLUEs, but strong prediction similarity generally limits uncertainty reduction.A small nonzero λy can explain more uncertainty on LSAT, Wine, and COMPAS; on MNIST, some settings also lower classification error, an effect not observed on other datasets.Source: §H.1 · p. 27, Figure 22 · p. 27, Figure 23 · p. 28
  • For deterministic neural networks, CLUE can explain away noise uncertainty, with more uncertain inputs requiring larger ℓ1 changes.These models cannot capture model uncertainty.Source: §H.1 · p. 27, Figure 24 · p. 28

H.2 VERIFYING RESULTS FROM OUR COMPUTATIONAL EVALUATION FRAMEWORK

  • On real data, CLUE outperforms U-FIDO on dNN−2 for every dataset except Wine, where results are very similar, and also performs better on the informativeness-to-relevance ratio.The gap is most pronounced on high-dimensional MNIST and Credit, where CLUE explains away more uncertainty with similarly close counterfactuals; sensitivity can reduce uncertainty but go off-manifold.Source: §H.2 · p. 29, Table 6 · p. 29, Table 7 · p. 29
  • Similar informativeness–relevance results on real data and artificial data suggest the computational evaluation framework is unbiased.The real-data comparison measures uncertainty reduction and distance to a training-set nearest neighbor because the real generative process is unavailable.Source: §H.2 · p. 29

H.3 ADDITIONAL ANALYSIS OF USER STUDY

  • In the tabular user-study breakdown, CLUE outperforms all baselines on both datasets, and its accuracy on certain test points almost doubles the next-best method’s.Methods perform similarly on uncertain points, while sensitivity does significantly worse on higher-dimensional COMPAS.Source: §H.3 · pp. 29–30, Table 8 · p. 30

I ADDITIONAL DETAILS ON THE GENERATIVE MODEL USED IN THE PROPOSED COMPUTATIONALLY GROUNDED EVALUATION FRAMEWORK

  • A two-level VAE addresses latent distribution mismatch: a second VAE models samples from the first VAE’s latent space, making the inner aggregate posterior closer to its prior.Sampling from the auxiliary latent space and mapping back through the two stages yields clean digits and restores ancestral sampling for the artificial-data experiments.Source: I ADDITIONAL DETAILS ON THE GENERATIVE… · p. 31, Figure 26 · p. 31, Figure 27 · p. 31

I.1 COMPARISON OF METHODS UNDER A GROUND TRUTH DGM

The authors argue that their synthetic-data evaluation can distinguish method performance without strongly favoring auxiliary-VAE methods. The additional survey details describe how certain and uncertain test points were selected and how survey participation was maintained.

I.1 COMPARISON OF METHODS UNDER A GROUND TRUTH DGM

  • The two-level VAEAC setup yields noticeable performance differences among evaluated methods, addressing concern that synthetic data might be too undifferentiated.The authors also argue that the VAE-based ground-truth generator gives CLUE and U-FIDO equal advantage because both use auxiliary DGMs.Source: §I.1 · p. 32
  • The largest CLUE–U-FIDO difference occurs on the ΔHgt versus Δlog pgt metric, which depends most on the ground-truth VAEAC.The authors note that the ground-truth model captures inputs and targets, whereas auxiliary DGMs model inputs.Source: §I.1 · p. 32

J.1 ADDITIONAL DETAILS ON TABULAR USER STUDY

  • Pilot participants were not told that test-point pools were divided by certainty, and the study retained at least ten participants for each of four survey variants.Three people exited without consent; one result was voided after an attention check, without changing the minimum completed-survey count.Source: §J.1 · pp. 32–33

J.2 MNIST USER STUDY

In a modified MNIST setting where rare digit features prompted high BNN uncertainty, practitioners who saw CLUEs classified new points’ uncertainty more accurately than those who saw labels alone.

J.2 MNIST USER STUDY

  • The modified dataset retained fours, sevens, and nines but removed rare dashed sevens and closed-top fours from training, producing high BNN epistemic uncertainty on those test features.No digits were deleted from the test set.Source: §J.2 · p. 33
  • Test digits exceeding the BNN uncertainty threshold were selected as uncertain examples or questions; Figure 30 shows CLUE examples for a four and a seven.The figure caption describes examples with characteristics uncommon in the modified MNIST dataset and their corresponding CLUEs and ∆CLUEs.Source: §J.2 · p. 33, Figure 30 · p. 34
  • The 10-question study compared label-only context against the same context augmented with CLUEs for uncertain points, using five randomly selected training examples as context.In both variants, practitioners judged whether new test points would be certain or uncertain to the BNN.Source: §J.2 · p. 34
  • Practitioners with CLUEs reached 88% accuracy on new test points, versus 67% for practitioners given context points and rejection labels alone.Each variant was shown to five graduate students with machine-learning expertise.Source: §J.2 · p. 34
Loading 2006.06848v2…