PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Evidential Regression

Alexander Amini, Wilko Schwarting, Ava Soleimany, Daniela Rus

arXiv:1910.02600v2cs.LGcs.NEstat.ML

TL;DR

The paper addresses the need for calibrated uncertainty in regression without requiring predefined OOD data or inference-time sampling. It learns evidential distributions for deterministic neural networks, achieving calibrated uncertainty across vision tasks while exposing regularization and prior-selection limitations.

  • Problem

    Regression uncertainty methods need calibrated OOD uncertainty from in-distribution training data, while continuous regression lacks a well-defined evidential regularization distance.

  • Method

    The method learns evidential distributions over Gaussian regression likelihoods to estimate aleatoric and epistemic uncertainty without sampling or OOD training data.

  • Results

    The method demonstrates calibrated uncertainty on benchmark and complex vision regression tasks, including OOD and adversarial robustness evaluations; evidential calibration error is 0.033 versus 0.048 for ensembling.

  • Takeaways & Limitations

    The approach provides a scalable, broadly applicable route to uncertainty estimation for regression across temporal forecasting, property prediction, control learning, and vision tasks.

  • Takeaways & Limitations

    The main limitations are tuning the regularization coefficient, removing non-misleading evidence during calibration, and determining the effects of alternative variance priors.

Abstract

from arXiv · show

Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such that the model is regularized when its predicted evidence is not aligned with the correct output. Our method does not rely on sampling during inference or on out-of-distribution (OOD) examples for training, thus enabling efficient and scalable uncertainty learning. We demonstrate learning well-calibrated measures of uncertainty on various benchmarks, scaling to complex computer vision tasks, as well as robustness to adversarial and OOD test samples.

1 Introduction

The paper addresses uncertainty estimation for deterministic regression networks used in safety-critical settings. It proposes learned evidential distributions to estimate both aleatoric and epistemic uncertainty without sampling or OOD training data.

  • Safety-critical regression applications require calibrated uncertainty to interpret confidence, detect OOD inputs, and recognize likely model failure.
  • Aleatoric uncertainty describes data uncertainty, whereas epistemic uncertainty describes uncertainty in the prediction.
  • Existing evidential methods primarily target classification and rely on uncertain-prior distances or OOD training, while regression lacks a well-defined regularization distance.
  • The proposed approach trains a network to predict higher-order evidential-distribution parameters over likelihood parameters such as (µ, σ2).
  • The method learns both uncertainty types without sampling during inference or training on OOD data, and evaluates calibration, robustness, and benchmark performance.

2 Modelling uncertainties from data

Deterministic regression commonly predicts an average target and may model aleatoric uncertainty through a Gaussian likelihood, but it does not capture predictive epistemic uncertainty. The paper places higher-order evidential priors over likelihood parameters to estimate both forms of uncertainty.

  • Squared-error regression encourages a model to learn the average correct answer but does not explicitly model noise or uncertainty in the data.
  • Maximum likelihood regression assumes targets arise from a Gaussian with mean and variance parameters θ = (µ, σ2), optimizing negative log likelihood.
  • Learning Gaussian likelihood parameters models aleatoric uncertainty but leaves the model oblivious to predictive epistemic uncertainty.
  • The proposed method directly learns evidence for predictions by placing higher-order prior distributions over parameters governing the observation distribution.
  • An evidential distribution represents confidence over likelihood parameters, and its realizations can generate lower-order likelihoods over observed data.

3 Evidential uncertainty for regression

The method places a Normal Inverse-Gamma evidential prior over Gaussian regression likelihoods and trains neural networks to infer its hyperparameters, representing both uncertainty types without inference sampling. Training combines model-evidence maximization with an error-scaled regularizer that reduces misleading evidence.

  • Evidential distribution: The Normal Inverse-Gamma distribution approximates the posterior over unknown Gaussian mean and variance through factorized Gaussian and Inverse-Gamma components.The prior is parameterized by m = (γ, υ, α, β), with γ ∈ R, υ > 0, α > 1, and β > 0.
  • Evidential distribution: Its hyperparameters determine likelihood location and dispersion, while total evidence Φ = 2υ + α counts inferred virtual observations.Increasing υ and α concentrates the evidential distribution around its inferred likelihood function.
  • Neural prediction: A neural network maps each input to evidential hyperparameters, enabling simultaneous regression, aleatoric uncertainty, and epistemic uncertainty estimation without OOD training data or sampling.The maximum-likelihood Gaussian is computed analytically, and epistemic uncertainty is obtained from the variance of the inferred evidential distribution.
  • Learning objective: Training maximizes model evidence by marginalizing the Gaussian likelihood parameters under the NIG prior, yielding an analytically evaluable objective.With a NIG prior, the marginal likelihood has an analytical solution and is expressed through a Student-t distribution.
  • Learning objective: A second objective minimizes evidence on incorrect predictions by scaling an error penalty with total inferred evidence.The regularizer addresses the undefined KL divergence to a zero-evidence NIG prior and avoids penalizing evidence everywhere in the unbounded regression output space.
  • Learning objective: The combined loss trades off model fit against uncertainty inflation, with λ = 0 producing over-confidence and excessively large λ causing over-inflation.The network outputs four parameters per target and uses constrained activations for υ, α, and β.

4 Experiments

Experiments compare evidential regression with established uncertainty methods on toy, benchmark, depth-estimation, OOD, and adversarial settings. The results show competitive predictive performance, calibrated uncertainty, and increasing uncertainty under distribution shift and perturbations.

  • Benchmark regression: On benchmark regression tests, evidential models outperform dropout sampling and ensembling on NLL and inference speed across all datasets.RMSE remains competitive despite the loss not explicitly optimizing accuracy.
  • Monocular depth estimation: Depth estimation scales evidential learning to over 27k indoor RGB-to-depth pairs using a U-Net-style neural network tested on disjoint scenes.The task predicts depth at every pixel, making the target high-dimensional.
  • Toy uncertainty estimation: In toy cubic regression, regularized evidential regression enables precise in-distribution predictions and conservative epistemic uncertainty where training data is absent.The comparison includes Gaussian MLE and ensembling baselines, with and without evidential regularization.
  • Monocular depth estimation: Evidential depth models localize prediction errors and produce confidence whose increasing levels correspond to steadily decreasing error.Dropout underestimates uncertainty, while ensembling occasionally overestimates it.
  • Monocular depth estimation: Evidential uncertainty achieves calibration error 0.033, outperforming ensembling at 0.048 and dropout at 0.126.Ideal calibration follows y = x.
  • OOD uncertainty: On unseen OOD driving images, evidential models show a positive entropy shift relative to in-distribution data, separating the uncertainty distributions.The evaluation uses entropy CDFs, interquartile comparisons, histograms, and sample predictions.
  • Adversarial robustness: Under increasing adversarial noise, evidential entropy rises and its spatial concentrations remain closely aligned with prediction error.The study uses FGSM perturbations and tracks error, epistemic uncertainty, entropy distributions, and pixel-wise visualizations.

5 Related work

The paper situates evidential regression relative to Bayesian neural networks, ensembles, dropout, evidential deep learning, and Prior Networks. Its distinguishing setup places uncertainty priors over predictive likelihoods and avoids sampling or OOD training data.

  • Prior networks and evidential models: Prior evidential and classification methods either require regularization toward a fixed prior, OOD training data, or are limited to discrete classification problems.The paper frames continuous regression as lacking a well-defined distance measure for regularizing the inferred evidential distribution.
  • Bayesian deep learning: Bayesian neural networks place priors over weights and use sampling-based approximations, while ensembles and dropout require multiple sampled inferences to estimate predictive variance.These approaches are described as computationally expensive compared with a single-pass deterministic approach.

6 Conclusions, limitations, and scope

The paper presents evidential regression as a scalable approach for joint uncertainty estimation and reports calibration, OOD, and vision-task results. It identifies regularization and evidence removal as central limitations, while calling for further study of variance priors.

  • Conclusions and scope: The method places evidential priors over regression likelihood outputs to learn both epistemic and aleatoric uncertainty.The paper positions this as a novel approach for regression problems.
  • Conclusions and scope: The authors demonstrate scalability to complex vision tasks, calibrated uncertainty on OOD data, and broad applicability across regression tasks.Examples include temporal forecasting, property prediction, and control learning.
  • Limitations and future work: The primary limitations are tuning the regularization coefficient and effectively removing non-misleading evidence during uncertainty calibration.The authors suggest dual optimization and alternative variance priors as directions for further investigation.

Broader Impact

The paper emphasizes calibrated uncertainty for neural networks deployed in safety-critical decision systems. It also warns that stronger model performance and uncertainty estimation may increase human trust, motivating continued redundancy.

  • Societal importance: Uncertainty estimation is intended to help interpret confidence, detect OOD domain shift, and recognize when a model is likely to fail in safety-critical applications.The motivation spans computer vision, robotics, control, and downstream decision systems.
  • Societal importance: Superior calibration over baselines is presented as supporting trust in algorithms and understanding when they indicate uncertainty.The paper frames this as complementary to improving neural-network precision and accuracy.
  • Societal challenges: Greater model performance and uncertainty capabilities may increase human trust, so the authors recommend redundancy to help catch and correct mistakes independently.This is identified as a potential societal challenge of uncertainty-aware systems.

S1 Derivations

The derivations model unknown Gaussian mean and variance with a Normal-Inverse-Gamma distribution, whose moments provide prediction and aleatoric and epistemic uncertainty. The KL-based zero-evidence regularizer is not directly evaluable, so the paper uses a small-evidence alternative but finds it highly sensitive.

  • The model assumes observations arise from a Gaussian with unknown mean and variance, then places a prior joint distribution over those parameters.
  • The first-order moments of the evidential distribution represent the maximum-likelihood prediction and both aleatoric and epistemic uncertainty.
  • Marginalizing the Normal-Inverse-Gamma parameters yields a Student-t posterior predictive distribution used to compute the negative log-likelihood loss.
  • The KL divergence between an arbitrary Normal-Inverse-Gamma distribution and a strictly zero-evidence distribution cannot be evaluated because the relevant Gamma function or division is undefined.
  • A small positive evidence constant makes the KL divergence well-defined but introduces a hyper-sensitive parameter.
  • The direct evidence regularizer was used instead because KL regularization was extremely sensitive to the small-evidence setting and failed to achieve the desired effect across datasets.

S2 Benchmark regression tasks

Synthetic regression experiments examine uncertainty estimation, regularization, and disentanglement of aleatoric and epistemic uncertainty. The evidential regularizer controls OOD epistemic uncertainty while preserving aleatoric uncertainty, and separates high noise from epistemic uncertainty in-distribution.

  • S2.1 Experimental setup: The cubic regression benchmark trains on y = x3 + ϵ with ϵ ∼N(0, 3) for −4 ≤ x ≤ 4 and tests over a wider range.
  • S2.1.3 Impact of the evidential regularizer: Reducing the evidential regularizer weight causes OOD uncertainty to decay toward zero, whereas stronger regularization inflates it while maintaining constant aleatoric uncertainty.
  • S2.1.4 Disentangled uncertainties: A synthetic dataset with increased central data noise suggests that the method disentangles in-distribution aleatoric uncertainty from epistemic uncertainty.
  • S2.1.3 Impact of the evidential regularizer: The experiments compare the evidential regularizer with KL-divergence regularization across several regularization coefficients λ.

S3 Depth estimation evaluations

Depth-estimation experiments evaluate evidential models against dropout and ensembles on predictive accuracy, efficiency, calibration, and robustness. The evidential method maintains competitive predictive performance while using fewer parameters and faster sampling-free inference, with uncertainty increasing on OOD and adversarial inputs.

  • S3.1 Depth estimation setup: Evidential depth models use a U-Net backbone and output four maps corresponding to γ, υ, α, and β.
  • S3.1 Depth estimation setup: Models are trained with Adam for 60,000 iterations, and evidential models use λ = 0.1 with three random initializations.
  • S3.2 Depth estimation performance metrics: The evidential method uses significantly fewer trainable parameters and faster forward-pass inference than ensembles while maintaining comparable RMSE and NLL.
  • S3.3 OOD uncertainty: Evidential uncertainty shows a positive entropy-CDF shift on OOD images and clear separation between ID and OOD uncertainty distributions.
  • S3.4 Aleatoric uncertainty estimation on depth: Evidential and Gaussian-likelihood aleatoric estimates strongly agree in reflection and poor-illumination scenes, identifying mirror reflections and dark regions without visible geometry as highly uncertain.
  • S3.4 Aleatoric uncertainty estimation on depth: The paper reports calibrated uncertainty across benchmarks, high-dimensional vision tasks, and new OOD and adversarial test samples without sampling during inference or OOD training data.
  • Limitations: The method has no guarantee of definitively producing high epistemic uncertainty far from in-domain regions.
Loading 1910.02600v2…