Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation

Janis Postels; Francesco Ferroni; Huseyin Coskun; Nassir Navab; Federico Tombari

Source-linked AI summary

Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation

Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, Federico Tombari

arXiv:1908.00598v3Aug 1, 2019cs.LGstat.ML

TL;DR

Safety-critical neural-network deployment requires uncertainty estimates that are available in real time, but MC-based epistemic estimation becomes expensive as sample counts grow. This paper approximates noise-injection uncertainty through sampling-free variance propagation, applying it to segmentation and depth regression. The approach is evaluated for uncertainty quality and computational efficiency across these visual tasks.

Problem
MC-based epistemic uncertainty estimation scales linearly with inference-time sample count, limiting its use in real-time and compute-constrained applications.
Method
The paper uses sampling-free variance propagation to approximate uncertainty from noise-injection methods, with a CNN-ReLU simplification for efficiency.
Results
The approximation is evaluated against sampling-based approaches on semantic segmentation and monocular depth estimation for uncertainty quality and computational overhead.
Takeaways & Limitations
The framework provides computationally efficient uncertainty maps for two large-scale computer-vision tasks, with better numerical approximation when noise layers are closer to the output.
Takeaways & Limitations
Full covariance propagation has O(N^2) memory complexity, and the ENCDEC setup underestimates output variance because it assumes a diagonal covariance matrix.

Abstract

from arXiv · show

We present a sampling-free approach for computing the epistemic uncertainty of a neural network. Epistemic uncertainty is an important quantity for the deployment of deep neural networks in safety-critical applications, since it represents how much one can trust predictions on new data. Recently promising works were proposed using noise injection combined with Monte-Carlo sampling at inference time to estimate this quantity (e.g. Monte-Carlo dropout). Our main contribution is an approximation of the epistemic uncertainty estimated by these methods that does not require sampling, thus notably reducing the computational overhead. We apply our approach to large-scale visual tasks (i.e., semantic segmentation and depth regression) to demonstrate the advantages of our method compared to sampling-based approaches in terms of quality of the uncertainty estimates as well as of computational overhead.

1. Introduction

Neural-network uncertainty is needed for safety-critical, real-time deployment, but sampling-based epistemic estimates can be computationally expensive. The paper proposes a sampling-free approximation for noise-injection methods and evaluates it across visual tasks.

Motivation: Real-time uncertainty quantification is required when neural networks support safety-critical applications such as autonomous driving or surgical robotics.The uncertainty must be available during inference so safety-critical systems can exploit it.
Motivation: Epistemic uncertainty concerns uncertainty in model parameters and helps indicate how much a model’s predictions can be trusted on unfamiliar data.This differs from aleatoric uncertainty, which arises from noise inherent in the data.
Related work: Noise injection and stochastic regularization estimate epistemic uncertainty by using prediction variance generated during inference.MC dropout is a prominent example that keeps dropout active at inference time.
Problem: MC sampling makes computation scale linearly with the number of samples, creating a real-time bottleneck for autonomous vehicles, robots, and mobile devices.The resulting cost can make MC dropout difficult for performance-critical or compute-limited applications.
Contribution: The paper proposes a sampling-free approximation of noise-injection uncertainty estimates and a CNN-ReLU simplification for computational efficiency.The work compares approximation quality with Bayesian SegNet on CamVid, tests withheld-class detection, and applies the method to monocular depth estimation.

3. Method

The method propagates activation uncertainty through noise layers, affine transformations, and nonlinearities to approximate epistemic uncertainty in a single forward pass. For convolutional ReLU networks, a diagonal-covariance simplification reduces computational cost but introduces assumptions about activation correlations and nonlinear transformations.

Error propagation: The method treats injected neural-network noise as activation errors and uses error propagation to estimate epistemic uncertainty in a single shot.It targets uncertainty estimates for networks trained with noise injected at inference time.
Assumptions and limitations: The approximation is limited by omitted activation correlations and by first-order treatment of nonlinear functions, while full covariance propagation can be prohibitively expensive for image-sized problems.The framework explicitly does not model the impact of correlations between activations on activation means.
Noise layers: Noise layers inject uncertainty through either additive or element-wise multiplicative noise, with covariance determined from the input covariance and noise covariance.The framework assumes independent noise across noise-layer nodes, while input activations may be dependent.
Affine layers and nonlinearities: Affine layers transform covariance exactly using the weight matrix, whereas nonlinearities are approximated with a first-order Taylor expansion.The activation-function Jacobians used include ReLU, sigmoid, and softmax; the softmax approximation error is analyzed separately.
Computational simplification: Propagating the full covariance matrix has O(N^2) memory complexity, so the diagonal approximation reduces variance propagation to the computational level of ordinary forward propagation.Under this assumption, only the covariance matrix diagonal is propagated, using element-wise squared weights or Jacobian entries.
Convolutional layers and ReLU: For convolutional layers followed by ReLU, the method assumes diagonal covariance because local connectivity does not transport substantial covariance across wide spatial regions.This approximation is motivated by sparse off-diagonal covariance after local convolutions and by variance dropping through ReLU.

4. Experiments

Experiments show that the sampling-free approximation produces uncertainty estimates similar to sampling-based methods across synthetic regression, semantic segmentation, and depth regression, while reducing inference-time computational cost. Its accuracy is strongest when the covariance approximation is appropriate, with identified deviations for distant dropout layers and sigmoid outputs.

Synthetic Data: The synthetic regression experiment shows increasing predictive standard deviation outside the training distribution, and the approximation is exact for this example.The sampling-based variance estimate converges toward the analytic estimate with many samples.
Predictive Performance: The approximation returns test log-likelihoods that are almost always marginally lower than or larger than MC dropout across regression datasets.Naval Propulsion is the exception, where the Gaussian assumption or MC sampling accuracy may affect TLL despite an RMSE of 0.00 ± 0.00.
Classification Task: Bayesian SegNet: The ENCDEC architecture can underestimate variance because its last dropout layer is far from the output and the method assumes diagonal covariance.This setup is nevertheless used to compare uncertainty quality with Bayesian SegNet.
Classification Task: Bayesian SegNet: Both methods identify similar high-uncertainty regions in semantic segmentation, especially around object boundaries.The approximated uncertainty magnitude is much lower than MC sampling, with a mean absolute difference of 93.7% of the MC mean variance.
Classification Task: Bayesian SegNet: The approximation has constant runtime without sampling, whereas MC dropout runtime increases linearly with the number of samples.Caching before the first dropout layer optimizes the sampling-based comparison, but the approximation retains the computational advantage.
Classification Task: Bayesian SegNet: Pixel misclassification rates vary similarly across uncertainty quantiles for the approximation and MC dropout, implying similar uncertainty calibration.The comparison uses quantiles because the two methods have different uncertainty magnitudes.
Regression Task: Depth Regression: In depth regression, both uncertainty estimates increase linearly with absolute prediction error, enabling identification of regions likely to have large regression errors.The paper reports this behavior with minimal computational overhead; sigmoid outputs prevent exact agreement with sampling-based estimates.

5. Conclusion

The framework uses error propagation to approximate sampling-based epistemic uncertainty estimates. Across two large-scale computer-vision tasks, it provides computational efficiency and coherent uncertainty maps, with better numerical approximation when noise layers are closer to the output.

Error propagation can approximate epistemic uncertainty estimates produced by training-time noise injection and inference-time sampling.
The approximation was applied to two large-scale computer-vision tasks, producing computationally efficient and coherent uncertainty maps.
The approximation is numerically better when the noise layer is located closer to the output layer.

Supplementary Material for ”Sampling-free Epistemic Uncertainty

The supplementary material develops the mathematical components underlying the sampling-free framework and adds comparisons, convergence evidence, and qualitative experiment results.

The supplement provides Jacobians for approximately propagating covariance through nonlinearities.
It derives covariance formulas for element-wise products and the expectation and variance of ReLU under a Gaussian distribution.
Additional experiments compare Monte-Carlo dropout with similar-runtime sampling and show convergence toward the analytic form on synthetic data.
The supplement includes further qualitative experiment results and refers to a supplementary video for qualitative results.

7. Jacobians of Activation Functions

The paper specifies Jacobians for the activation functions used in its experiments: ReLU, sigmoid, and softmax.

For ReLU, the derivative at the origin is assumed to be zero, yielding a Jacobian that is one when i = j and x_i > 0, and zero otherwise.
The element-wise sigmoid function has a separately specified Jacobian for propagating covariance through the activation.
The softmax Jacobian is expressed using softmax outputs S_i and S_j together with the Kronecker delta δ_ij.

8. Covariance of Hadamard Product of Random Vectors

This section derives the covariance of the element-wise product of two independent random vectors, using the covariance definition and independence assumption to obtain the desired form.

The derivation considers Y⃗ = Z⃗ ◦ X⃗, the element-wise product of independent random vectors Z⃗ and X⃗.
Independence permits factorizing expectations in the covariance calculation for the Hadamard product.
The derivation compares the expanded expectation expression with the covariance definition to obtain the desired covariance form.

9. Expectation and Variance of ReLU Given a Gaussian Distribution

For a Gaussian input, the paper derives the expectation and variance of ReLU under a diagonal-covariance assumption, supporting analytic uncertainty propagation without sampling.

9. Expectation and Variance of ReLU Given a Gaussian Distribution: The section derives first- and second-order moments for ReLU applied to a scalar normally distributed variable.The analysis restricts inputs to scalars because it assumes a diagonal covariance matrix.
9. Expectation and Variance of ReLU Given a Gaussian Distribution: For X ~ N(µ, σ), the expectation of ReLU is computed from its defining integral.
9. Expectation and Variance of ReLU Given a Gaussian Distribution: The error function is used to express the Gaussian integral underlying the ReLU expectation.
9. Expectation and Variance of ReLU Given a Gaussian Distribution: The variance is obtained from the second moment and the squared expectation of the ReLU output.
9. Expectation and Variance of ReLU Given a Gaussian Distribution: Figure 10 compares mean and standard-deviation absolute differences against ground truth as the sample count changes, with the proposed method remaining constant without sampling.

10. Comparison with Monte-Carlo (MC) Dropout of Similar Computational Cost

On Boston Housing, the proposed approximation is computationally advantageous beyond approximately 175 samples and remains more accurate than MC dropout for standard-deviation estimation through 500 samples.

10. Comparison with Monte-Carlo (MC) Dropout of Similar Computational Cost: Approximately 175 samples marks the point beyond which the proposed method becomes computationally advantageous.
10. Comparison with Monte-Carlo (MC) Dropout of Similar Computational Cost: Even with up to 500 samples, MC dropout does not match the accuracy of the proposed standard-deviation approximation.

11. Absolute Variance Difference for Synthetic Data

On synthetic data, the sampling-based variance estimate converges to the paper’s analytic approximation as the number of samples increases.

11. Absolute Variance Difference for Synthetic Data: The sampling-based variance estimate converges to the analytic expression for large numbers of samples.
11. Absolute Variance Difference for Synthetic Data: The relative absolute difference between the sampling-based estimate and the approximation converges to zero as sample count grows.

12. Qualitative Results Including Our Prediction

Qualitative evaluations show similar uncertainty patterns for the sampling-based and approximate methods across segmentation, depth regression, and withheld-class experiments, while revealing task-specific uncertainty locations.

12.1. Bayesian SegNet [18]: On CamVid segmentation, the network is mostly uncertain around object boundaries.
12.2. Monocular Depth Regression [12]: On KITTI monocular depth regression, predictions are most certain in the highest-resolution depth region and uncertain near the image’s left and right borders.The border uncertainty is attributed to training with non-overlapping stereo images.
12.3. Qualitative Results of Class Hold-out: When pedestrians and cyclists are withheld during training, their locations tend to receive higher uncertainty than other image regions.
12. Qualitative Results: The qualitative comparisons include original inputs, ground truth or predictions, MC-dropout uncertainty, and the proposed approximation across the evaluated tasks.