PaperlayerView exact revision on arXiv

Source-linked AI summary

DEUP: Direct Epistemic Uncertainty Prediction

Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Ion Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, Yoshua Bengio

arXiv:2102.08501v4cs.LGstat.ML

TL;DR

Variance-based uncertainty can miss lack of knowledge caused by model misspecification. DEUP estimates excess risk with a secondary error predictor, subtracting aleatoric uncertainty when available. The paper reports advantages from added features in interactive settings and more useful exploration estimates in a toy GP example.

  • Problem

    Variance- or entropy-based measures omit model uncertainty caused by misspecification, despite its contribution to a predictor’s excess risk.

    Source: §1 · p. 2

  • Method

    DEUP trains a secondary error predictor to estimate pointwise generalization risk, then subtracts estimated aleatoric uncertainty when available or provides an upper bound otherwise.

    Source: §1 · p. 2

  • Results

    The paper reports experimental advantages for added features that address interactive error-target non-stationarity, while its toy example shows more useful estimates than a misspecified GP.

    Source: §1 · p. 3, §6 · p. 16

  • Takeaways & Limitations

    DEUP’s uncertainty estimates can support purposeful exploration by interactive learners, with the framework agnostic to predictor type and search method.

    Source: §1 · p. 2

    The zero-aleatoric-noise shortcut applies when the data-generating process is noiseless; the paper uses this condition in its Section 5.1 experiments.

    Source: §3 · p. 7

Abstract

from arXiv · show

Epistemic Uncertainty is a measure of the lack of knowledge of a learner which diminishes with more evidence. While existing work focuses on using the variance of the Bayesian posterior due to parameter uncertainty as a measure of epistemic uncertainty, we argue that this does not capture the part of lack of knowledge induced by model misspecification. We discuss how the excess risk, which is the gap between the generalization error of a predictor and the Bayes predictor, is a sound measure of epistemic uncertainty which captures the effect of model misspecification. We thus propose a principled framework for directly estimating the excess risk by learning a secondary predictor for the generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. We discuss the merits of this novel measure of epistemic uncertainty, and highlight how it differs from variance-based measures of epistemic uncertainty and addresses its major pitfall. Our framework, Direct Epistemic Uncertainty Prediction (DEUP) is particularly interesting in interactive learning environments, where the learner is allowed to acquire novel examples in each round. Through a wide set of experiments, we illustrate how existing methods in sequential model optimization can be improved with epistemic uncertainty estimates from DEUP, and how DEUP can be used to drive exploration in reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic image classification and predicting synergies of drug combinations.

1 Introduction

The introduction argues that variance-based uncertainty misses model-misspecification bias, and presents DEUP as an excess-risk estimate for interactive learning settings where data and error targets change.

1 Introduction

  • Epistemic uncertainty can inform active learning, sequential model optimization, and reinforcement-learning exploration by indicating where additional information may be valuable.It is a measure of lack of knowledge that an active learner should minimize, unlike irreducible aleatoric randomness.Source: §1 · p. 2
  • Variance- or entropy-based uncertainty misses model uncertainty from misspecification, which can leave a predictor confidently wrong.The paper frames model uncertainty as part of excess risk: the gap between a predictor’s risk and the Bayes predictor’s risk.Source: §1 · p. 2
  • DEUP estimates epistemic uncertainty by training an error predictor for pointwise generalization risk and subtracting estimated aleatoric uncertainty when available.Without an aleatoric estimate, the framework provides an upper bound on epistemic uncertainty.Source: §1 · p. 2
  • In the toy example, DEUP maps underestimated GP variances to L2 errors, producing more useful exploration estimates than the misspecified GP.The GP standard deviation fails to enclose the ground function in [0.5, 1.5].Source: §1 · p. 3, Figure 1 · p. 3
  • Interactive exploration can shift the input distribution and change error-prediction targets, so DEUP adds features informative about both the input and current dataset.Density and model-variance estimates are among the features used, and may sometimes come at no additional cost.Source: §1 · p. 3

2 Excess Risk, Epistemic Uncertainty, and Model Misspecification

The section frames excess risk as epistemic uncertainty because it captures both finite-data uncertainty and model misspecification, which posterior dispersion alone can miss. It explains why this bias matters in practice, including interactive learning.

2.1 Notations and Background

  • In supervised learning, predictors map inputs to actions and are assessed by expected loss; learners use finite data to find a low-risk function within a hypothesis space.The hypothesis space may be tuned using a validation subset, affecting the learner’s effective capacity.Source: §2.1 · p. 4

2.2 Sources of lack of knowledge

  • Epistemic uncertainty includes model uncertainty from restricting the hypothesis space and approximation uncertainty from finite data and computational resources.The Bayes predictor’s remaining risk reflects irreducible aleatoric uncertainty rather than reducible lack of knowledge.Source: §2.2 · p. 5
  • Excess risk—the point-wise gap between a predictor’s risk and the Bayes predictor’s risk—captures epistemic uncertainty from both model and approximation uncertainty.The two components can be reduced by improving the hypothesis space and by obtaining more data, respectively.Source: §2.2 · p. 5

2.3 Bayesian uncertainty under model misspecification

  • Posterior variance or entropy can indicate approximation uncertainty but omit bias; under misspecification, Bayesian credible sets may over-cover or under-cover.Bayesian posteriors may concentrate around the best model available within the hypothesis space without fitting the ground-truth distribution.Source: §2.3 · p. 6
  • Practical misspecification can arise even when a model is theoretically well specified, because priors, finite computation, regularization, or early stopping restrict reachable predictors.In interactive learning, acquiring examples where bias is large can change the effective hypothesis space and reduce bias and excess risk; relying on posterior variance alone may slow learning.Source: §2.3 · p. 7

3 Direct Epistemic Uncertainty Prediction

DEUP estimates epistemic uncertainty by predicting point-wise generalization error and subtracting estimated aleatoric uncertainty. For interactive learning, it trains the error predictor on out-of-sample errors from earlier predictors and uses features to account for changing training data.

3 Direct Epistemic Uncertainty Prediction

  • DEUP trains an error predictor on observed out-of-sample errors to estimate point-wise risk, then subtracts estimated aleatoric uncertainty to estimate excess risk.When aleatoric uncertainty cannot be estimated, the predicted risk can serve as a conservative proxy for epistemic uncertainty.Source: §3 · pp. 7–8

3.1 Fixed Training Set

  • With a fixed training set, a held-out validation set supplies out-of-sample errors for training the secondary predictor of point-wise risk.The estimated aleatoric uncertainty is subtracted from its output to obtain the uncertainty estimate.Source: §3.1 · p. 8

3.2 Interactive Settings

  • In interactive learning, the error predictor must estimate risk as a function of both the input and the changing training dataset, not the input alone.Future acquired points serve as out-of-sample examples for earlier predictors; using only in-sample errors can limit generalization and underestimate uncertainty.Source: §3.2 · p. 9
  • To avoid storing every version of the main predictor, DEUP embeds input–dataset pairs as stationarizing features and uses each acquired point once to form error-training data.The feature set can include input, data density, training-set membership, and model variance; experiments report that stationarizing features are critical for reliable estimates, while including the input can help or hurt across tasks.Source: §3.2 · pp. 9–10
  • When interactive data are too scarce to train the error predictor promptly, its training set can be pre-filled from the initially available examples using a strategy inspired by K-fold cross-validation.The described procedure targets a small multiple of the number of initial training points.Source: §3.2 · p. 10
  • DEUP’s interactive procedure is agnostic to the acquisition machinery, which proposes new inputs using the current predictor and its uncertainty estimate.The stopping rule is determined by the interactive task.Source: §3.2 · p. 10

4 Related Work

Related work covers Bayesian and ensemble uncertainty estimates, distribution-free conformal prediction, and methods that predict loss or uncertainty directly. DEUP differs from cited loss-prediction approaches by targeting out-of-sample error relevant to active learning.

Bayesian Learning

  • Bayesian methods represent epistemic uncertainty through posterior distributions, while ensembles and related methods use dispersion or disagreement among predictors as uncertainty signals.The section also reviews evidential methods and deep-kernel approaches that combine learned representations with Bayesian inference.Source: §4 · pp. 10–11

Distribution-Free Uncertainty Estimation

  • Conformal prediction provides prediction sets or intervals with specified coverage guarantees and can be applied to existing models without a special training procedure.The section characterizes DEUP as distribution-free but distinguishes it from conformal prediction because DEUP does not require a pre-defined confidence degree for a prediction set.Source: §4 · p. 12

Loss Prediction

  • Prior loss-prediction methods estimate in-sample error or ensemble variance, whereas DEUP targets out-of-sample error relevant to selecting inputs with large reducible generalization error.The cited work also includes binary methods that distinguish low from high epistemic uncertainty.Source: §4 · p. 12

5 Experiments

The experiments evaluate DEUP in sequential model optimization and reinforcement learning, and test whether its uncertainty estimates generalize in drug-combination prediction and OOD classification.

5 Experiments

  • The experiments test downstream decision gains and uncertainty generalization without additional OOD training data; DEUP training with stationarizing features is on par with training an ensemble of five networks.Source: §5 · p. 12

5.1 Sequential Model Optimization

  • On the Ackley benchmark, DEUP-EI consistently reaches the 10-D global maximum while GP-EI gets stuck, and TuRBO-DEUP-EI outperforms TuRBO-EI, especially at higher dimensions.On the synthetic function, GP-EI requires more than 50 steps on average to reach the global maximum.Source: Figure 4 · p. 13, §5.1 · p. 13, Figure 3 · p. 13

5.2 Reinforcement Learning

  • On CartPole, DEUP-DQN achieves lower regret faster than DQN with ϵ-greedy, DQN with MC-Dropout, and Bootstrapped DQN.DEUP uncertainty is trained to predict TD-error and used as an exploration bonus in Q-values.Source: §5.2 · p. 14, Figure 5 · p. 14

5.3.1 Epistemic Uncertainty Estimation for Drug Combinations

  • For drug-combination synergy prediction, DEUP’s out-of-sample errors correlate better with test residuals and capture their magnitude, while ensembles and MC-dropout underestimate uncertainty.The experiments use DrugComb and LINCS L1000 data.Source: §5.3.1 · p. 14, Figure 6 · p. 15

5.3.2 Epistemic Uncertainty Predictions for Rejecting Difficult Examples

  • On OOD data, DEUP significantly outperforms baselines in SRCC and matches Deep Ensembles on AUROC while scoring better than other methods.The main predictor achieves 93.89% accuracy; the authors report no explicit trade-off between its accuracy and uncertainty-estimate quality.Source: §5.3.2 · p. 15, Table 1 · p. 16

6 Conclusion and Future Work

The paper argues that variance-based epistemic uncertainty misses model uncertainty from misspecification, and proposes excess risk as a measure that captures it. DEUP estimates predictor risk with a second model and addresses non-stationarity in interactive settings with additional input features.

6 Conclusion and Future Work

  • Variance-based measures capture approximation uncertainty but miss bias from model misspecification, which the paper treats as part of epistemic uncertainty.The paper identifies excess risk—the gap between a predictor’s risk and the Bayes predictor’s risk—as a measure that includes model uncertainty.Source: §6 · p. 16
  • DEUP trains a second model to predict the main predictor’s risk and uses an estimate of aleatoric uncertainty to estimate excess risk.In interactive settings, the risk estimator faces non-stationarity, which the authors address with extra input features.Source: §6 · p. 16
  • The authors identify computational efficiency, aleatoric-uncertainty estimation without repeated oracle queries, and theoretical analysis under misspecification as future-work directions.They also mention exploring different stationarizing features or ways to incorporate the dataset into the secondary predictor.Source: §6 · p. 16

A Sequential Model Optimization Experiments

The sequential optimization experiments compare acquisition strategies and describe how DEUP’s uncertainty estimator is trained. In the one-dimensional experiment, GP-EI performed worse than random acquisition on the function with multiple local maxima.

A Sequential Model Optimization Experiments

  • The uncertainty-estimator dataset was pre-filled by alternating in-sample and out-of-sample roles across halves of the initial data, yielding four training points per initial point.The procedure used two random splits; out-of-sample examples supplied high-epistemic-uncertainty instances.Source: A Sequential Model Optimization… · p. 23
  • The error predictor used log MSE targets because error scales varied across multiple orders of magnitude.Source: A Sequential Model Optimization… · p. 23
  • The sequential optimization experiments used BoTorch and took about one CPU day in total.Training time varied by optimization step and depended on feature, input-dimension, and learning-algorithm choices.Source: A Sequential Model Optimization… · p. 23

A.1 One-dimensional function

  • On the one-dimensional function, GP-EI required more optimization steps and performed worse than random acquisition.The target function was designed with multiple local maxima.Source: §A.1 · p. 23
  • For the one-dimensional experiment, GP variance was the stationarizing input; binary in-sample/out-of-sample and density features added no improvement in function calls.A neural-network uncertainty estimator gave similar results but cost more computationally in this low-data setting.Source: §A.1 · p. 23

A.2 Two-dimensional function

On the two-dimensional Levi N.13 benchmark, DEUP-EI consistently reached the global maximum in fewer than 56 function evaluations. The experiment used a domain with multiple local maxima.

A.2 Two-dimensional function

  • The authors used Levi N.13, a known two-dimensional optimization benchmark, to test DEUP’s sequential model optimization performance.Source: §A.2 · p. 24
  • The Levi N.13 experiment optimized over [−10, 10]², where the maximum is 0 at (1, 1).The function has multiple local maxima.Source: §A.2 · p. 24
  • DEUP-EI was the only method reported to reach the global maximum consistently in under 56 function evaluations.The comparison concerns sequential optimization on Levi N.13.Source: §A.2 · p. 24, Figure 7 · p. 24

A.3 Additional details for the Ackley function experiment, for synthetic data in higher dimensions

In the higher-dimensional synthetic-function experiments, DEUP calibrates a Gaussian process’s variance to predict out-of-sample squared error. The authors report that adding the input x to the stationarizing feature harmed performance through overfitting.

A.3 Additional details for the Ackley function experiment, for synthetic data in higher dimensions

  • The Ackley benchmark’s sole global optimizer is (0, …, 0), where the function value is 0; the experiments used BoTorch’s default A, B, and c values of 20, 0.2, and 2π.Source: §A.3 · p. 24
  • The Ackley experiments used the domain [−10, 15]^d, and the TurBO baseline used Expected Improvement with batch size 1.Source: §A.3 · p. 24
  • DEUP used a Gaussian process as the main model and another GP to calibrate its variance to out-of-sample squared error.The uncertainty predictor used log targets and log variances as inputs; TurBO-DEUP applied calibration to TurBO’s local GP models.Source: §A.3 · pp. 24–25
  • A feature-subset experiment trained DEUP-EI for 50 iterations from six initial pairs and compared the maximum values reached across five runs.The figure reports distributions at iteration 40 and mean maximum-value trajectories for feature subsets that passed a 0.9 threshold.Source: Figure 8 · p. 25
  • Adding x alongside the stationarizing feature made the GP error predictor overfit and hurt final performance.Source: §A.3 · p. 25

A.4 Ablation study for the stationarizing features

The ablation study finds that stationarizing features beyond the input help the error predictor, while including the input can improve interactive-learner performance.

A.4 Ablation study for the stationarizing features

  • Stationarizing features beyond the input are important to the error predictor’s performance.The study compares feature subsets used as input to the error predictor.Source: §A.4 · p. 25
  • Including the input among the features can improve the performance of an interactive learner.Source: §A.4 · p. 25
  • The ablation compares subsets of the “xdvb” features as inputs to the error predictor.These represent the input, density estimate, variance estimate, and a bit indicating whether the input was part of the training set.Source: §A.4 · p. 25

B Reinforcement Learning Experiments

The reinforcement-learning experiments use bsuite to evaluate DQN-based agents with metrics for sample complexity and final performance. DEUP-DQN combines a Q-network with a density-based uncertainty estimator.

B Reinforcement Learning Experiments

  • The experiments use bsuite’s basic metric and average regret to assess sample complexity and final performance.Source: B Reinforcement Learning Experiments · p. 25
  • The base agent is DQN with a three-layer fully connected Q-network, alongside Bootstrapped DQN and DQN with MC-Dropout baselines.The MC-Dropout baseline uses two dropout layers with probability 0.1 before the second and third fully connected layers.Source: B Reinforcement Learning Experiments · p. 25
  • For action selection, the agent takes a stochastic Q-network forward pass, equivalent to sampling Q-values from a posterior as in Thompson sampling.Source: B Reinforcement Learning Experiments · p. 25
  • The KDE is fit every 10,000 actions using replay-buffer states, and the uncertainty estimator has the Q-network’s layer count plus a final Softplus layer.The replay buffer has capacity 10,000.Source: B Reinforcement Learning Experiments · pp. 25–26
  • DEUP-DQN’s uncertainty estimator takes the log-density of states as input, with state densities estimated by a kernel density estimator.The algorithm specifies a KDE fitting frequency and a number of warm-up episodes.Source: B Reinforcement Learning Experiments · p. 26

C Rejecting Difficult Examples

On CIFAR-10-C, DEUP’s uncertainty estimates correlate better with prediction error than the baselines across corruptions. Feature ablations also find that combining features performs best on OOD data.

C Rejecting Difficult Examples

  • For the OOD-rejection experiments, uncertainty was assessed by its Spearman correlation with generalization error and by OOD-detection AUROC.DEUP uses log-density, model-variance, and seen/unseen features for its error predictor; the density estimator is a masked autoregressive flow and the variance estimator is DUE.Source: C Rejecting Difficult Examples · p. 26
  • The error predictor is trained using examples from two held-out CIFAR-10 classes in each fold, with log MSE targets for errors across multiple scales.Five folds each use eight classes to train the main predictor and the remaining two as OOD examples for error-predictor training.Source: C Rejecting Difficult Examples · pp. 26–27
  • The variance feature alone yields a Spearman correlation of 37.84 ± 0.04, log-density alone yields 30.52 ± 0.03, and the image alone yields 36.58 ± 0.16.Source: C Rejecting Difficult Examples · p. 27
  • Combining all features performs best; using log-density and variance as error-predictor features outperforms using them directly, and the seen-example bit improves performance.The compared inputs include log-density, variance, and a bit indicating whether an example was seen.Source: C Rejecting Difficult Examples · p. 27, Table 6 · p. 28

C.1 Predicting Uncertainty under Distribution Shift

  • DEUP’s uncertainty estimates correlate better with prediction error than the baselines on CIFAR-10-C, across all corruption types.Figure 9 reports Spearman rank correlation for models trained on CIFAR-10 and evaluated on CIFAR-10-C.Source: §C.1 · p. 28, Figure 9 · p. 28

D Drug Combination Experiments

In drug-combination experiments, DEUP’s out-of-sample uncertainty estimates are more consistent with residual magnitudes than those of MC-Dropout, ensembles, and DUE. The experiments assess uncertainty for predicting drug-combination synergy using DrugComb and LINCS L1000 data.

D Drug Combination Experiments

  • The experiments use DrugComb anti-cancer compound pairs and LINCS L1000 gene-expression profiles, restricting pairs to those with profiles for both drugs in PC-3 and MCF-7 cell lines.Each drug is represented by a 1,024-bit Morgan fingerprint and two 978-dimensional gene-expression profiles.Source: D Drug Combination Experiments · p. 29
  • Uncertainty methods are compared on combination-sensitivity prediction for 1,385 PC-3 examples using a 40/30/30 train, validation, and test split.The underlying predictor sums the two drugs’ embeddings before predicting synergy, making it invariant to their order.Source: D Drug Combination Experiments · p. 29
  • The evaluation reports residual–uncertainty correlation and its estimated upper bound, plus predictive log-likelihood, 68% interval coverage, and interval width.The reported metrics are averaged over three seeds.Source: Table 7 · p. 30, D Drug Combination Experiments · p. 31
  • DEUP’s out-of-sample uncertainties are more consistent with residual magnitudes, while MC-Dropout, ensembles, and DUE underestimate uncertainty.Figure 10 visualizes predicted means and uncertainties for 50 test examples ordered by true synergy score.Source: D Drug Combination Experiments · p. 31, Figure 10 · p. 32
  • DUE’s predicted uncertainties have a narrow range, which yields a lower upper bound for correlation with residuals than other methods.A similar pattern was observed with other kernels in the DUE package, including the standard Gaussian kernel.Source: D Drug Combination Experiments · p. 31
  • Replicates of a drug-combination experiment could estimate aleatoric uncertainty, allowing it to be subtracted from out-of-sample uncertainty to leave epistemic uncertainty.Source: D Drug Combination Experiments · p. 31

E DEUP in the presence of aleatoric uncertainty

With Gaussian-noise training data, GP variance conflates epistemic and aleatoric uncertainty, and DEUP's noisy error targets make out-of-sample data especially important for estimating total uncertainty.

E DEUP in the presence of aleatoric uncertainty

  • With Gaussian-noise training data, the gap between GP-variance EU and true EU is higher than in the deterministic setting.True EU is measured by the mean squared error between the GP mean and the noiseless ground-truth function.Source: E DEUP in the presence of aleatoric… · p. 32
  • An aleatoric-uncertainty estimate learned from noisy training data can be subtracted from DEUP's error predictor to estimate epistemic uncertainty.The experiment uses a simple linear regressor to estimate aleatoric uncertainty and avoid overfitting.Source: E DEUP in the presence of aleatoric… · p. 32
  • Because DEUP's error-prediction targets are noisy in this setting, reasonable total-uncertainty estimates require more out-of-sample data.Source: E DEUP in the presence of aleatoric… · p. 32
  • Figure 11's lower-right panel uses additional out-of-sample data in low-density regions to train a second GP to predict the first GP's generalization error.The predicted generalization error represents total uncertainty.Source: Figure 11 · p. 33
  • DEUP's outputs are unconstrained and can predict negative uncertainties, so practical use calls for soft clipping.Source: E DEUP in the presence of aleatoric… · p. 33

Related papers

Loading 2102.08501v4…