PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Deterministic Uncertainty: A Simple Baseline

Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, Yarin Gal

arXiv:2102.11582v3cs.LGstat.ML

TL;DR

Reliable uncertainty estimation seeks computationally cheaper alternatives to methods requiring multiple forward passes or expensive approximate inference. The paper tests whether well-regularized feature spaces suffice, introducing DDU, which combines post-training GDA feature density with a softmax network. DDU performs as well as Deep Ensembles and outperforms SNGP and DUQ across reported OoD detection and active-learning settings.

  • Problem

    Existing uncertainty methods can require multiple forward passes, expensive Monte-Carlo sampling, or substantial memory and computation, while predictive uncertainty conflates epistemic and aleatoric uncertainty.

  • Method

    DDU trains a residual model with spectral normalization, then fits GDA post-training for feature-space epistemic uncertainty while retaining the softmax distribution for aleatoric uncertainty.

  • Results

    DDU performs as well as Deep Ensembles and outperforms SNGP and DUQ on several OoD benchmarks and active-learning settings.

  • Takeaways & Limitations

    DDU provides a simple alternative to Deep Ensembles for producing epistemic and aleatoric uncertainty estimates without their reported complexity or computational cost.

  • Takeaways & Limitations

    Feature-space density can fail when feature extractors map OoD inputs to in-distribution regions, and some uncertainty designs have unresolved aleatoric-epistemic limitations.

Abstract

from arXiv · show

Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive. We take two complex single-forward-pass uncertainty approaches, DUQ and SNGP, and examine whether they mainly rely on a well-regularized feature space. Crucially, without using their more complex methods for estimating uncertainty, a single softmax neural net with such a feature-space, achieved via residual connections and spectral normalization, *outperforms* DUQ and SNGP's epistemic uncertainty predictions using simple Gaussian Discriminant Analysis *post-training* as a separate feature-space density estimator -- without fine-tuning on OoD data, feature ensembling, or input pre-procressing. This conceptually simple *Deep Deterministic Uncertainty (DDU)* baseline can also be used to disentangle aleatoric and epistemic uncertainty and performs as well as Deep Ensembles, the state-of-the art for uncertainty prediction, on several OoD benchmarks (CIFAR-10/100 vs SVHN/Tiny-ImageNet, ImageNet vs ImageNet-O) as well as in active learning settings across different model architectures, yet is *computationally cheaper*.

1. Introduction

The paper asks whether computationally complex single-pass uncertainty methods need more than a well-regularized feature space. It introduces DDU, combining spectral-normalized feature representations, post-training GDA density, and softmax predictions to estimate and disentangle epistemic and aleatoric uncertainty.

  • Motivation: Deep Ensembles often perform best in uncertainty prediction, but their training and test-time memory and compute burden limits adoption.Deterministic single-forward-pass methods aim to reduce this burden.
  • Method: DDU fits GDA after training to estimate epistemic uncertainty from feature-space density, without OoD hold-out data, feature ensembling, or input preprocessing.Spectral normalization supplies the feature-space regularization used by the approach.
  • Method: DDU combines GDA-based epistemic uncertainty with a cross-entropy-trained softmax distribution for aleatoric uncertainty and in-distribution calibration.Temperature scaling can be applied to the original softmax layer after GDA fitting.
  • Results: DDU performs as well as Deep Ensembles and outperforms SNGP and DUQ on several OoD benchmarks and active learning settings.The comparison requires no model-architecture changes beyond spectral normalization.
  • Disentangling uncertainty: A well-regularized feature space separates in-distribution Dirty-MNIST from OoD Fashion-MNIST, whereas LeNet and VGG-16 feature densities overlap because of feature collapse.Softmax entropy separates ambiguous from unambiguous in-distribution samples but does not reliably distinguish in-distribution from OoD data.
  • Caveats: Feature-space density alone can fail when a feature extractor maps OoD inputs into in-distribution regions, making feature-space regularization a central scope condition.The paper focuses on spectral normalization in common residual architectures and uses GDA as a simple density estimator.

2. Background

The background distinguishes epistemic uncertainty, which decreases with more data, from aleatoric uncertainty, which reflects irreducible ambiguity or noise. It motivates feature-space density methods and explains why predictive uncertainty and deterministic feature representations can fail to separate these sources reliably.

  • Epistemic uncertainty is high for previously unseen inputs and decreases when those inputs are added to training data.
  • Aleatoric uncertainty is high for ambiguous or noisy in-distribution samples and does not decrease with additional data.
  • Predictive entropy can be high for both ambiguous in-distribution and out-of-distribution inputs, whereas ensemble mutual information is selective for out-of-distribution uncertainty.
  • Approximate Bayesian methods can be difficult to scale, computationally expensive, or limited in uncertainty quality.
  • Feature-space distance and density estimate epistemic uncertainty relative to training data, but feature collapse can map out-of-distribution inputs into in-distribution regions.
  • Bi-Lipschitz regularization combines input sensitivity with feature smoothness to help prevent feature collapse and preserve robustness.

3. Deep Deterministic Uncertainty

DDU uses spectral normalization and residual connections to regularize a deterministic model's feature space, then fits GDA after training for epistemic uncertainty while retaining softmax entropy for aleatoric uncertainty. It achieves performance comparable to Deep Ensembles in active learning without their full computational burden.

  • DDU fits GDA after training to estimate epistemic uncertainty from feature-space density without requiring OoD data or additional preprocessing steps.
  • Spectral normalization with residual connections encourages feature sensitivity and smoothness while preserving model accuracy.
  • DDU combines feature-space density for epistemic uncertainty with the softmax predictive distribution for aleatoric uncertainty.
  • DDU performs on par with Deep Ensembles in active learning experiments.
  • Softmax entropy provides aleatoric uncertainty for in-distribution samples and can be further calibrated using temperature scaling.

4. Experiments

DDU is evaluated for active learning and OoD detection across datasets and architectures, where it generally matches or exceeds strong deterministic and ensemble baselines.

  • Active Learning: DDU reaches 90% MNIST accuracy with 160 training samples, versus 185 for Deep Ensemble PE and 155 for Deep Ensemble MI.DDU is three times faster than the three-model Deep Ensemble during acquisition.
  • Active Learning: On Dirty-MNIST, DDU reaches 70% accuracy with 240 training samples, while the deterministic softmax baseline peaks at 50%.DDU also outperforms SNGP, DUQ, VGG-16 feature density, and the Deep Ensemble PE baseline in this setting.
  • Active Learning: DDU’s feature density and Deep Ensemble MI avoid confusion from in-distribution ambiguous samples with high aleatoric uncertainty.This advantage grows as the training-set size increases in Dirty-MNIST.
  • OoD Detection: DDU outperforms DUQ, SNGP, and energy-based deterministic methods across the reported OoD dataset pairs and often performs on par with Deep Ensembles.The pattern holds across the evaluated architectures, including ImageNet versus ImageNet-O, without compromising single-model test accuracy.
  • OoD Detection: Wide-ResNet-28-10 with spectral normalisation outperforms its corresponding ensemble in almost all reported cases, unlike VGG-16 feature density.The comparison supports the importance of residual connections and spectral normalisation for feature-density uncertainty.

5. Additional Insights

The paper analyzes how predictive entropy and feature-density models can misrepresent uncertainty because aleatoric and epistemic objectives are not generally aligned.

  • Predictive Entropy: Predictive entropy can be high for both ambiguous in-distribution samples and OoD samples, making it unreliable for OoD detection on datasets with ambiguity.It can separate in-distribution and OoD data better on curated datasets with low aleatoric uncertainty.
  • Objective Mismatch: Conditional log-likelihood, marginalised log-likelihood, and joint log-likelihood generally have different optima in a noisy mixture model.The synthetic example uses three classes with 4% label noise.
  • Softmax Entropy: A deterministic softmax model may not preserve the epistemic-uncertainty ordering captured by a Deep Ensemble’s mutual information.The proposition shows that a more epistemically uncertain sample can receive lower softmax entropy from some ensemble member.
  • Objective Mismatch: A one-component-per-class mixture model cannot generally optimize both feature-space density and predictive-distribution estimation objectives.When a GMM does optimize both, it must be a GDA, although the converse need not hold.
  • DDU: DDU combines a softmax classifier for aleatoric uncertainty with a separate feature-density estimator for epistemic uncertainty.The softmax is trained with conditional log-likelihood, while the feature-density estimator is fitted separately.

6. Conclusion

DDU is presented as a simple, lower-complexity alternative for estimating aleatoric and epistemic uncertainty, with scope limited to supervised feature-space regularization and GDA density estimation.

  • Conclusion: DDU combines post-training GDA feature-density estimation with a residual, spectrally normalized neural network to estimate epistemic and aleatoric uncertainty.Its feature-density approach is supervised and relies on feature-space regularization rather than contrastive pretraining.
  • Conclusion: DDU can outperform deterministic single-pass uncertainty methods and perform as well as Deep Ensembles in active learning and OoD detection.The conclusion frames it as an alternative without the complexity or computational cost of current state-of-the-art methods.
  • Scope and Related Work: Earlier feature-density approaches may underperform because of feature collapse, objective mismatch, input perturbation requirements, or OoD fine-tuning.The cited related-work passage specifically notes that competitive Lee et al. results require input perturbation.
  • Scope and Related Work: DDU does not require additional hyperparameters beyond minimal changes to the standard softmax setup or training and fine-tuning with OoD data.Its GMM parameters are optimized for the already trained model using the training set.

C.2. Algorithm & Pseudo-Code Implementation

The implementation trains a spectrally normalized model, fits density statistics after training, and uses density and softmax entropy jointly to classify uncertainty.

  • Thresholding: OoD thresholds can be obtained from in-distribution training data, with points below the 99% density quantile labeled OoD.This thresholding procedure does not require OoD training examples.
  • Implementation: Training uses SGD with momentum 0.9 for 50 epochs, with learning-rate drops by a factor of 10 at epochs 25 and 40.Online spectral normalisation uses one power-iteration step on convolutional weights, with coefficient 3.
  • Algorithm: The pseudo-code trains the neural network on labeled data before estimating class-wise Gaussian-mixture parameters from the training set.The GMM is constructed after the classifier has been trained.
  • Algorithm: For a sample, DDU computes its feature representation, evaluates GMM density, and computes softmax entropy.Low density is classified as OoD; high-density samples are separated into ambiguous in-distribution or ordinary in-distribution cases by entropy.

D.2. OoD Detection Training Setup

The experiments use standard training setups for softmax and ensemble baselines, while SNGP and DUQ follow their published configurations across several GPU-trained architectures.

  • Softmax baselines and the 5-Ensemble train for 350 epochs with SGD, momentum 0.9, and learning-rate drops at epochs 150 and 250.
  • SNGP and DUQ use the training setups and hyperparameters specified in their respective papers.
  • Models for the large-scale tasks are trained on a single Nvidia Quadro RTX 6000 GPU, while toy-study models use a single Nvidia GeForce RTX 2060 GPU.

E. Additional Results

Additional OoD experiments and ablations examine DDU across architectures, datasets, corruption types, and uncertainty objectives, emphasizing the role of feature-space inductive biases.

  • Additional OoD results: Additional OoD results cover CIFAR-10 and CIFAR-100 dataset pairs across ResNet-50, ResNet-110, DenseNet-121, Wide-ResNet-28-10, and VGG-16.
  • Corruption analysis: The corruption analysis reports AUROC against corruption intensity for Wide-ResNet-28-10, ResNet-50, ResNet-110, and DenseNet-121 on CIFAR-10-C.
  • Ablations: The ablations compare DDU with and without inductive biases, sensitivity alone, and a 5-Ensemble across OoD detection settings.
  • Ablation findings: Feature density without the proposed inductive biases yields significantly lower AUROC than Wide-ResNet-28-10 with inductive biases and does not outperform the corresponding ensemble on any dataset.
  • Ablation findings: The small AUROC difference between Wide-ResNet models with and without spectral normalization suggests residual connections may contribute more than smoothness constraints.
  • Uncertainty decomposition: DDU uses softmax output for aleatoric uncertainty and GMM feature density for epistemic uncertainty, allowing post-hoc softmax calibration without changing feature densities.

F.1.1. DISENTANGLING EPISTEMIC AND ALEATORIC UNCERTAINTY

The motivational experiments distinguish epistemic uncertainty from aleatoric uncertainty by comparing softmax entropy and feature-space density on ambiguous in-distribution and OoD data.

  • Dirty-MNIST combines unambiguous digits with ambiguous samples, creating a setting where uncertainty methods must distinguish data ambiguity from distributional novelty.
  • AUROC measures epistemic uncertainty quality, whereas ECE measures aleatoric uncertainty quality in the Dirty-MNIST evaluations.
  • 99.9% AUROC is achieved only by fitting a GMM over a feature extractor with appropriate inductive biases, while softmax entropy captures aleatoric but not epistemic uncertainty.
  • On Two Moons, DDU is uncertain away from the data distribution, unlike single-model softmax entropy and ensemble predictive entropy, which are uncertain mainly near the decision boundary.
  • Feature-collapse analysis explains why OoD features mapped into in-distribution regions can receive densities indistinguishable from in-distribution samples.

F.3. Two Moons

The Two Moons and training-set-size experiments test whether DDU’s feature density tracks epistemic uncertainty while softmax entropy tracks aleatoric uncertainty.

  • Two Moons: The Two Moons study compares softmax entropy, three-model predictive entropy, and spectrally normalized DDU feature density using residual and non-residual fully connected networks.
  • Two Moons: DDU is confident only on the data distribution, whereas softmax and ensemble entropies are uncertain primarily along the class decision boundary.
  • Two Moons: Overlapping density histograms do not necessarily imply feature collapse because other causes can produce the same overlap.
  • Training-set-size experiments: With larger training sets, feature-space density increases consistently with decreasing epistemic uncertainty and correlates strongly with ensemble mutual information.
  • Objective mismatch ablation: GMM feature density performs better for epistemic uncertainty, while softmax performs better for aleatoric uncertainty, including accuracy and ECE.
  • Training-set-size experiments: Softmax entropy remains roughly unchanged as training data increases, consistent with aleatoric uncertainty being irreducible and independent of training-set size.

F.6. Objective Mismatch Ablation with Wide-ResNet-28-10 on CIFAR-10

The ablation examines why softmax entropy and feature-space density optimize different uncertainty objectives. It shows that each estimator excels on its own target, while GDA provides a compromise that is weaker for aleatoric uncertainty.

  • Ablation result: Feature-space density performs worse than the softmax layer for aleatoric uncertainty on Wide-ResNet-28-10 and CIFAR-10.The comparison uses accuracy and ECE.
  • Softmax-entropy mismatch: Deep Ensembles can exhibit higher epistemic uncertainty while individual deterministic models show lower softmax entropy at similar predictive entropy.Proposition 5.2 formalizes this mismatch under a gap δ and entropy difference bounded by ϵ.
  • Analytical framework: Dirichlet modeling is used only as an analysis tool to characterize distributions of softmax probability vectors, not as a fitted component of the method.The analysis requires strong assumptions and does not fit Dirichlet distributions to data.
  • Analytical result: The entropy variance analysis empirically provides a lower bound on the true variance across model-parameter softmax entropies.This connects the theoretical analysis to the variability expected across deterministic model components.
  • DDU design: DDU combines a softmax classifier for predictive and aleatoric uncertainty with a post-training GDA density estimator for epistemic uncertainty.The two components optimize different objectives rather than forcing one model to serve both roles.

G.2.2. PROOF

The proof establishes that discriminative classification and marginal feature-density estimation optimize distinct objectives. A shared optimum is possible only under restrictive conditions, and generally the objectives trade off.

  • Objective definitions: A discriminative classifier is optimized by conditional log-likelihood, whereas a density estimator is optimized by marginal log-likelihood.The proposition identifies these as separate objectives for p(y | z) and q(z).
  • Shared optimum: A one-component-per-class GMM may fail to maximize conditional and marginal log-likelihood simultaneously; if it does, the resulting model must be GDA.The converse does not hold: a GDA need not maximize both objectives.
  • Objective trade-off: Generally, minimizing both objectives simultaneously is impossible because stationary points can involve a trade-off between conditional and marginal feature-space entropy.The proof relates the two objectives through joint entropy and shows that their optima need not coincide.
  • Model interpretation: For GMMs with one component per class, minimizing joint entropy corresponds to GDA, while minimizing marginal feature entropy corresponds to fitting a density estimator.The density-estimation objective does not constrain component assignments by class.
  • Scope of the result: With ambiguous data, perfect class separation is impossible, so a shared optimum is rare; EM yields a better density estimator and the softmax layer a better classifier than GDA.These conclusions apply to noisy real-world data under the stated GMM setting.

G.2.3. INTUITIONS & VALIDATION WITH A TOY EXAMPLE

A synthetic three-class example with 4% label noise illustrates that classification, joint modeling, and density estimation favor different solutions. The results motivate using softmax outputs for aleatoric uncertainty and GDA density for epistemic uncertainty.

  • Toy-example setup: 4% label noise causes the objectives to diverge in the three-class synthetic 2D example.Figure 6 visualizes the resulting differences among the optimization objectives.
  • Classification objective: The LDA-like softmax solution has the lowest label NLL and a wide decision boundary that captures aleatoric uncertainty.Overlapping Gaussian components are needed to represent ambiguous labels.
  • Objective comparison: Each solution minimizes its own objective best; the GMM is the best density model, the LDA the best label model, and GDA is nearly as good for density.Table 12 reports the realized objective scores, where smaller is better.
  • Joint objective: The GDA solution has a narrower decision boundary and higher NLL because it captures aleatoric uncertainty less effectively.Its untied covariance matrices can also produce curved decision boundaries and unexpected behavior far from the data.
  • Aleatoric uncertainty: An ambiguous point receives high entropy under the softmax-trained objective but very low entropy under GDA, making GDA overconfident.The example directly demonstrates the mismatch in aleatoric-uncertainty representation.
  • DDU interpretation: DDU therefore uses the softmax layer for classification and aleatoric uncertainty and GDA as the separate density model for epistemic uncertainty.The objectives do not align, so the two roles are assigned to different estimators.
  • Density and OoD behavior: With ResNet-18 and spectral normalization, in-distribution and out-of-distribution densities separate, whereas LeNet and VGG-16 exhibit overlapping densities from feature collapse.Softmax entropy captures aleatoric uncertainty for in-distribution data but can confound in-distribution and out-of-distribution points.
Loading 2102.11582v3…