Deep Deterministic Uncertainty: A Simple Baseline

• Deep Ensembles (Lakshminarayanan et al., 2017) generally lead uncertainty prediction but incur substantial memory and compute costs; DUQ and SNGP use specialized outputs and cannot disentangle both uncertainties.DUQ uses RBF outputs and a Jacobian penalty; SNGP (Liu et al., 2020a) uses GP outputs and spectral normalization, with limitations in its uncertainty estimates.Source: §1 · pp. 1–2
• DDU combines post-training GDA on spectrally normalized features for epistemic uncertainty with the trained softmax distribution for aleatoric uncertainty.The softmax is trained with cross-entropy as a proper scoring rule and can be temperature-scaled for in-distribution calibration; GDA requires no OoD hold-out training, feature ensembling, or input preprocessing.Source: §1 · p. 2
• DDU performs as well as Deep Ensembles on several OoD benchmarks and active-learning settings, while outperforming SNGP and DUQ without architectural changes beyond spectral normalization.The comparison is with Deep Ensembles’ epistemic uncertainty (Lakshminarayanan et al., 2017); the reported benchmarks include CIFAR-10/100 versus SVHN/Tiny-ImageNet and ImageNet versus ImageNet-O.Source: §1 · p. 2, Table 1 · p. 2
• Dirty-MNIST adds ambiguous digits with multiple plausible labels; softmax entropy captures their aleatoric uncertainty but confounds in-distribution and OoD samples and can vary arbitrarily across models for the same OoD input.The paper presents Dirty-MNIST to make this issue clearer than artificially curated datasets such as MNIST or CIFAR-10.Source: §1 · pp. 2–3, Figure 2 · p. 3
• Feature-space density needs a well-regularized feature space: without smoothness and sensitivity, it may fail to separate in-distribution from OoD data, unlike ResNet-18 with spectral normalization.The paper contrasts ResNet-18 with spectral normalization against LeNet and VGG-16, whose feature densities overlap for Dirty-MNIST and Fashion-MNIST.Source: §1 · p. 3, Figure 2 · p. 3
• The study focuses on spectral normalization in common residual architectures and uses GDA as a simple post-training density estimator, while noting that more complex estimators might improve density estimates.The authors contrast this scope with unsupervised contrastive methods requiring large datasets and density estimators requiring expectation-maximization or variational inference.Source: §1 · p. 3

Uncertainty concepts

• Epistemic uncertainty is high for previously unseen inputs and falls as they enter training, whereas aleatoric uncertainty reflects ambiguity or noise, does not fall with more data, and is meaningful only in-distribution.Figure 4 illustrates feature-space density increasing as training data grows while softmax entropy stays roughly unchanged.Source: §2 · p. 3, Figure 4 · p. 4

Bayesian Models; Deep Ensembles

• Bayesian methods quantify epistemic uncertainty through mutual information between parameters and outputs, but approximate inference can be unscalable, low-quality, or dependent on costly Monte Carlo sampling (Gal, 2016; Smith & Gal, 2018).Deep Ensembles average model softmax outputs and estimate uncertainty with the entropy of that average; despite high training and test costs, they remain state-of-the-art.Source: §2 · p. 4

Deterministic Models

• Softmax confidence and entropy are commonly used for deterministic models, but softmax entropy can be low or high on OoD inputs, and existing improvements may fail under distribution shift or require OoD training data (Hendrycks & Gimpel, 2016).In Figure 3, ensemble mutual information is high for OoD inputs, while predictive entropy is high for both OoD and ambiguous in-distribution regions.Source: §2 · p. 4, Figure 3 · p. 4

Feature-Space Distances and Density; Feature Collapse

• Feature-space distance and density can proxy epistemic uncertainty by estimating distance or density to training data: adding a previously unseen point shifts it from high distance/low density to low distance/high density, under feature-space assumptions.Feature collapse may map OoD inputs to iD regions, causing these estimates to fail out of the box (van Amersfoort et al., 2020; van Amersfoort et al., 2021).Source: §2 · p. 4

Smoothness & Sensitivity

• Bi-Lipschitz constraints promote feature sensitivity and smoothness; residual connections with spectral normalization encourage this property faster and more stably than Jacobian gradient penalties (Liu et al., 2020a).The lower Lipschitz bound preserves input-space distances, while the upper bound limits excessive feature sensitivity.Source: §2 · pp. 4–5

Ensuring Sensitivity & Smoothness

• Spectral normalization in residual models encourages feature sensitivity and smoothness, with minor ResNet changes intended to preserve accuracy.Source: §3 · p. 5

Disentangling Epistemic & Aleatoric Uncertainty

• After training, GDA fits one Gaussian component per class from training-feature means and covariances, requiring one training-set pass and no OoD data.Source: §3 · p. 5

Evaluation

• Feature marginal likelihood estimates epistemic uncertainty, while softmax entropy estimates aleatoric uncertainty for in-distribution samples.High feature density indicates low epistemic uncertainty, so softmax uncertainty and predictions can be trusted; low density indicates OoD and makes predictions untrustworthy.Source: §3 · p. 5
• In active learning, DDU performs on par with Deep Ensembles as acquired training-set size increases.Source: Figure 5 · p. 5

4.1. Active Learning

• On MNIST, DDU reaches 90% test accuracy at 160 training samples versus 245 for softmax; Deep Ensemble reaches 90% at 185 (PE) and 155 (MI).DDU is three times faster than a Deep Ensemble; on CIFAR-10, it also outperforms softmax and is competitive with Deep Ensembles.Source: §4.1 · p. 6
• On Dirty-MNIST, DDU reaches 70% test accuracy with 240 training samples, while softmax accuracy peaks at 50%.With a 1:60 ratio of MNIST to ambiguous samples, DDU and Deep Ensemble MI outperform the other baselines; DDU also exceeds Deep Ensemble PE, consistent with their epistemic scores avoiding confusion from ambiguous samples.Source: §4.1 · p. 6

4.2. OoD Detection

• Across the tested OoD benchmarks and architectures, DDU outperforms other deterministic methods, often matches or exceeds Deep Ensembles, and does not compromise single-model test accuracy.The evaluations include CIFAR-10 and CIFAR-100 dataset pairings, CIFAR-10-C, and ImageNet vs ImageNet-O.Source: §4 · p. 5, §4.2 · p. 7
• A VGG-16 feature-density model does not beat its ensemble, whereas Wide-ResNet-28-10 with spectral normalization outperforms its corresponding ensemble in almost all cases.The paper attributes the difference in part to spectral normalization’s bi-Lipschitz constraint and notes that residual connections may also contribute to good performance.Source: §4.2 · p. 7

Potential Pitfalls of Predictive Entropy

• Predictive entropy can reflect either ambiguous in-distribution samples or out-of-distribution samples, limiting OoD detection on datasets with ambiguous examples.It can indicate the other uncertainty well when either aleatoric or epistemic uncertainty is already known to be low; predictive entropy has outperformed mutual information in prior empirical findings (Malinin & Gales, 2018). Ensembling can be viewed as Bayesian Model Averaging (Wilson & Izmailov, 2020), allowing mutual information to disentangle epistemic uncertainty.Source: §5 · p. 7

Objective Mismatch

• Figure 6 illustrates that the best-calibrated model need not provide the best density estimate, and GDA can assign no aleatoric uncertainty to an ambiguous sample.The figure uses a three-component GMM on a synthetic three-class dataset with 4% label noise.Source: Figure 6 · p. 8

Potential Pitfalls of Softmax Entropy

• A deterministic model’s softmax entropy may fail to preserve epistemic-uncertainty rankings because ensemble members can make arbitrary predictive extrapolations.Proposition 5.2 shows that, for some ensemble members, the sample with higher ensemble mutual information has lower softmax entropy; OoD softmax entropy can therefore be high, low, or intermediate, though this behavior is not universal.Source: §5 · pp. 7–8

Objective Mismatch

• A classifier optimized for conditional log-likelihood and a feature-density estimator optimized for marginal log-likelihood generally cannot both be optimal in the same class-conditional mixture model.This objective mismatch was overlooked in prior deterministic-UQ work (Liu et al., 2020a; Postels et al., 2020). A one-component-per-class GMM that maximizes both objectives must be GDA, but GDA need not maximize both.Source: §5 · p. 8
• DDU uses a softmax classifier to capture aleatoric uncertainty for in-distribution samples and a separate feature-density estimator to capture epistemic uncertainty.The classifier is trained with conditional log-likelihood, the usual cross-entropy objective.Source: §5 · p. 8

6. Conclusion

• DDU performs as well as deep ensembles in various settings and can outperform deterministic single-pass uncertainty methods in active learning and OoD detection at lower computational cost.It fits a GDA for feature-space density estimation after training a model with residual connections and spectral normalization.Source: §6 · p. 8

A. Related Work

• Earlier density-based methods can fail to distinguish OoD from in-distribution samples or report weaker epistemic uncertainty when their feature spaces collapse or their estimators conflate uncertainty types.Postels et al. (2020) report 74% AUROC on CIFAR-10 vs SVHN; Lee et al. (2018b) require OoD fine-tuning, input perturbations, and density ensembling across layers for competitive results.Source: A. Related Work · p. 12
• DDU requires minimal changes from a standard softmax setup, avoids OoD-data training or fine-tuning, and uses supervised bi-Lipschitz inductive biases for reliable feature-space density estimation.The GMM parameters are optimized for the already trained model using the training set; this approach is orthogonal to Winkens et al. (2020), who use contrastive training.Source: A. Related Work · p. 12

A.1. Predictive Entropy and Confidence in Recent Works

• The recent-work survey finds that few papers examine mutual information with deep ensembles as an OoD score, while none address uncertainty confounding or softmax entropy and confidence consistency.The survey describes a selection of papers, not an exhaustive list.Source: §A.1 · p. 12

B. Ambiguous- and Dirty-MNIST

• Ambiguous-MNIST contains 60K training and 10K test images decoded from mixtures of two MNIST-digit latent representations, while Dirty-MNIST combines it with MNIST in a 1:1 ratio.Ambiguous-MNIST labels are sampled from an ensemble's softmax probabilities; entropy is stratified class-wise to preserve a range of ambiguity levels.Source: B. Ambiguous- and Dirty-MNIST · p. 13

C.1. Increasing sensitivity

• To preserve sensitivity during residual downsampling, the network replaces strided 1x1 convolutions with average pooling and uses leaky ReLUs so negative activations continue to propagate.The conventional strided 1x1 convolution depends only on each 2x2 patch's top-left pixel, reducing sensitivity; the architectural change improves uncertainty quality without sacrificing accuracy.Source: §C.1 · pp. 13–14

C. Algorithm

• DDU's uncertainty-disentangling procedure treats low feature density as OoD, then uses softmax entropy to distinguish ambiguous in-distribution inputs from ordinary in-distribution inputs.For high-density samples, high entropy indicates ambiguous in-distribution data and low entropy indicates in-distribution data.Source: §C.1 · p. 14

C.2. Algorithm & Pseudo-Code Implementation

• The procedure sets points below the training-set density 99% quantile as OoD, using in-distribution training data to determine density or entropy thresholds.The passage specifies the 99% quantile rule for density; it does not specify an equivalent quantile for entropy.Source: §C.2 · p. 14
• Listing 1 presents a simple Python pseudo-code implementation with a scikit-learn-like API (Buitinck et al., 2013).The algorithm itself is provided in Algorithm 1.Source: §C.2 · p. 14

D.1. Dirty-MNIST

• Training runs for 50 epochs with momentum 0.9, initial learning rate 0.1, and tenfold learning-rate drops at epochs 25 and 40.The learning-rate schedule and optimizer settings are specified for the experiments.Source: §D.1 · p. 14
• Online spectral normalization uses one power-iteration step on convolutional weights, with coefficient 3 selected by cross-validation.For 1x1 convolutions the exact algorithm is used; for 3x3 convolutions, the approximate algorithm from Gouk et al. (2021) is used.Source: §D.1 · p. 14
• The pseudo-code instantiates a sensitive smooth model and GDA, fits the model on loaded training samples and labels, and computes aleatoric uncertainty as prediction entropy.It obtains predictions by applying the model's softmax layer to test features.Source: §D.1 · p. 15
• The listing marks a sample OoD when its epistemic uncertainty is at least the OoD threshold.The supplied code expresses the test as `ood_threshold <= epistemic_uncertainty`.Source: §D.1 · p. 15
• Table 4 evaluates OoD detection with ResNet-50 across CIFAR-10 and CIFAR-100 dataset pairs, averaging results over 25 runs.The listed pairings include CIFAR-10 vs SVHN, CIFAR-100, and Tiny-ImageNet, and CIFAR-100 vs SVHN and Tiny-ImageNet; the supplied caption gives no metric values or winning method.Source: Table 4 · p. 15

D.2. OoD Detection Training Setup

• CIFAR-10/100 softmax baselines and the 5-Ensemble trained for 350 epochs with SGD, momentum 0.9, learning rate 0.1, and 10× drops at epochs 150 and 250.Source: §D.2 · p. 15
• SNGP and DUQ used the training setup and hyperparameters specified in their respective papers (Liu et al., 2020a).Source: §D.2 · p. 15
• ImageNet models trained for 90 epochs with SGD, learning rate 0.1, weight decay 1e-4, warmup decay 0.01, and a step scheduler with size 30 and factor 0.1.Source: §D.2 · p. 15

D.3. Compute Resources

• Models for large-scale active learning and the listed CIFAR-10/CIFAR-100 OoD tasks used a single Nvidia Quadro RTX 6000 GPU.The models included ResNet-18, Wide-ResNet-28-10, ResNet-50, ResNet-110, DenseNet-121, and VGG-16.Source: §D.3 · p. 15
• LeNet, VGG-16, and ResNet-18 models producing Figure 2 and Table 9 results each trained on a single Nvidia GeForce RTX 2060 GPU.Source: §D.3 · p. 15
• ImageNet models (ResNet-50, Wide-ResNet-50-2, and VGG-16) trained using 8 Nvidia Quadro RTX 6000 GPUs.Source: §D.3 · p. 15

E. Additional Results

• Additional OoD evaluations cover CIFAR-10 and CIFAR-100 dataset pairs across ResNet-50, ResNet-110, and DenseNet-121, with CIFAR-10-C corruption results across four architectures.Tables 4–6 report dataset-pair results, while Figures 8–11 show AUROC versus corruption intensity for individual CIFAR-10-C corruption types.Source: E. Additional Results · pp. 15–16
• VGG-16 feature density without the proposed inductive biases has lower AUROC than Wide-ResNet-28-10 with inductive biases and never beats its corresponding ensemble across the tested datasets.This supports the claim that GMM feature density alone cannot estimate epistemic uncertainty reliably in a model with feature collapse; sensitivity and smoothness conditions are needed.Source: Table 7 · p. 16, E. Additional Results · p. 19
• Residual connections appear to matter more than spectral normalization: removing spectral normalization makes only a minimal AUROC difference, although the normalized Wide-ResNet-28-10 still performs better.The passage attributes this pattern to residual connections imposing sensitivity, which can map OoD samples farther from in-distribution samples.Source: E. Additional Results · p. 19
• DDU uses softmax output for aleatoric uncertainty and GMM feature density for epistemic uncertainty, with temperature scaling able to calibrate softmax without affecting feature density.The passage states that feature densities capture epistemic uncertainty well, while post-hoc calibration can address softmax miscalibration.Source: E. Additional Results · p. 19

F. Toy Experiments & Additional Ablations

• Toy-experiment uncertainty quality is assessed using Dirty-MNIST test ECE for aleatoric uncertainty and Dirty-MNIST versus Fashion-MNIST AUROC for epistemic uncertainty.These metrics are identified as proxies in Table 9; the supplied passage does not report their values.Source: Table 9 · p. 20

F.1. Motivational Example in Figure 2

• On Dirty-MNIST, softmax entropy overlaps between Fashion-MNIST OoD samples and ambiguous in-distribution samples, whereas ResNet-18+SN feature density distinguishes OoD from in-distribution samples.LeNet and VGG-16 feature densities do not separate OoD from in-distribution samples in the described Figure 2 comparison.Source: §F.1 · p. 20
• DDU separates uncertainty types: low feature density indicates high epistemic uncertainty, while high feature density together with high softmax entropy indicates high aleatoric uncertainty.Dirty-MNIST includes ambiguous digits with multiple plausible labels and therefore varying aleatoric uncertainty.Source: §F.1 · p. 20

F.1.1. DISENTANGLING EPISTEMIC AND ALEATORIC UNCERTAINTY

• The uncertainty experiments use MNIST as unambiguous iD, Ambiguous-MNIST as ambiguous iD, and Fashion-MNIST as OoD, training LeNet, VGG-16, and ResNet-18 with spectral normalisation on Dirty-MNIST.Dirty-MNIST combines Ambiguous-MNIST and standard MNIST.Source: §F.1.1 · p. 20
• Around 84% AUROC: softmax entropy poorly distinguishes Dirty-MNIST from Fashion-MNIST, although all three models obtain similar ECE scores on the Dirty-MNIST test set.Figure 2(b) shows strong overlap between softmax entropy for OoD and ambiguous iD samples.Source: §F.1.1 · p. 20
• Around 71% and 89% AUROC: LeNet and VGG-16 feature-space densities, respectively, fail to distinguish OoD from iD samples without appropriate inductive biases.This shows that feature-space density alone is insufficient.Source: §F.1.1 · p. 20
• 99.9% AUROC: DDU’s GMM feature density with an appropriately biased feature extractor outperforms the ablation alternatives on Dirty-MNIST versus Fashion-MNIST.The feature extractor uses a ResNet with spectral normalisation; OoD samples are not seen during training.Source: §F.1.1 · p. 20
• Softmax entropy estimates aleatoric but not epistemic uncertainty, whereas feature density estimates epistemic uncertainty only with a sensitive, smooth feature extractor such as DDU’s ResNet with spectral normalisation.The softmax outputs capture aleatoric uncertainty even without additional inductive biases.Source: §F.1.1 · p. 20
• On the Two Moons dataset, DDU feature density is uncertain away from the data distribution, while softmax entropy and deep-ensemble predictive entropy are uncertain only along the decision boundary.A fully connected network without inductive biases does not capture uncertainty properly through feature density.Source: Figure 12 · p. 21

F.2. Effects of a well-regularized feature space on feature collapse

• With a regularized feature space, OoD and iD feature-density histograms do not overlap, indicating that OoD features remain farther from class centroids and feature collapse is absent.Without regularization, Figure 2(c) shows overlapping densities, consistent with some OoD features mapping onto iD features.Source: §F.2 · p. 21

F.3. Two Moons

• On Two Moons, DDU is confident only on the data distribution, while softmax entropy and ensemble predictive entropy are uncertain only near the class decision boundary.The setup uses 2,000 samples with noise rate 0.1 and compares a single softmax model, a 3-ensemble, and DDU.Source: §F.3 · pp. 21–22
• On Two Moons, feature-density uncertainty fails without residual connections, underscoring the need for suitable inductive biases; deep ensembles can also underperform on this simple dataset.The FC-Net ablation lacks residual connections, and the authors attribute ensemble underperformance to components generalizing in the same way.Source: §F.3 · p. 21
• The Two Moons results show DDU capturing uncertainty in a small 2D setup, while the density-histogram caveat is that overlap does not necessarily imply feature collapse.Histograms can overlap for reasons other than feature collapse.Source: §F.3 · pp. 21–22

F.4. 5-Ensemble Visualisation

• In the 5-ensemble visualization, individual softmax entropies vary across the out-of-distribution interval (−2, 2) despite model disagreement, whereas ensemble mutual information is high there.Softmax entropy is high in ambiguous-label regions between −4.5 and −3.5 and between 3.5 and 4.5, showing it can capture aleatoric uncertainty.Source: §F.4 · p. 22
• Ensemble predictive entropy is high both in the out-of-distribution interval and at ambiguous points, capturing epistemic and aleatoric uncertainty in the visualization.The ambiguous points are at −4 and 4.Source: §F.4 · p. 22

F.5. Feature-Space Density & Epistemic Uncertainty vs Softmax Entropy & Aleatoric Uncertainty

• As training data grows, feature-space density rises as epistemic uncertainty falls, and negative log density correlates strongly and consistently with ensemble mutual information on DirtyMNIST and CIFAR-10.The experiments use ResNet-18+SN models trained on increasingly large data subsets; Figure 13 and Table 10 present the results.Source: §F.5 · pp. 22–23, Figure 13 · p. 22, Table 10 · p. 22
• Softmax entropy stays roughly unchanged as training data grows, consistent with aleatoric uncertainty being irreducible and independent of training-set size.The authors note this pattern is consistent with experiments in Kendall & Gal (2017).Source: §F.5 · p. 23

F.6. Objective Mismatch Ablation with Wide-ResNet-28-10 on CIFAR-10

• On CIFAR-10 with Wide-ResNet-28-10, the feature-space density estimator performs worse than the softmax layer for aleatoric uncertainty, measured by accuracy and ECE.The passage reports this as an ablation validating Proposition 5.3 and points to Table 11.Source: §F.6 · p. 23

G.1.1. QUALITATIVE STATEMENT; G.1.2. INFINITE DEEP ENSEMBLE

• With similar predictive entropy, if one point has greater ensemble epistemic uncertainty by δ, a non-null set of ensemble members gives it lower softmax entropy by at least δ − ϵ.The ensemble parameter distribution can be interpreted as either a finite ensemble or a distribution over trained models, making ensemble members samples of that distribution.Source: §G.1.1 · pp. 23–24, §G.1.2 · p. 24

G.1.3. ANALYSIS OF SOFTMAX ENTROPY OF A SINGLE DETERMINISTIC MODEL ON OOD DATA USING PROPERTIES OF DEEP ENSEMBLES

• Under strong assumptions, a maximum-entropy Dirichlet model estimates softmax-entropy variability from predictive distribution and expected information gain, but real variance is much higher on OoD samples.This analysis treats the Dirichlet as a modeling tool, not a distribution to fit to the data, and concludes that softmax entropy poorly captures epistemic uncertainty on high-uncertainty samples.Source: §G.1.3 · p. 24

Preliminaries; 3. The expectation E

• Dirichlet preliminaries derive moments of log probabilities and the categorical entropy, including its variance, for the later softmax-entropy analysis.The appendix states the expectation and covariance of log probabilities and gives results for expected categorical entropy and its variance.Source: Preliminaries · p. 24, §3 · pp. 26–27

G.2. Objective Mismatch

• The objectives for aleatoric and epistemic uncertainty do not match: DDU uses the softmax layer as a discriminative classifier for predictive entropy and a GMM as a generative classifier for feature-space density.The CIFAR-10 ablation reports that the feature-space density estimator performs worse than the softmax layer for aleatoric uncertainty.Source: §F.6 · p. 23, §G.2 · p. 28

Main Result; Empirical Results

• The Dirichlet variance approximation lower-bounds empirical softmax-entropy variance, while fitted Dirichlet samples approximate the empirical entropy distribution; Figure 15 reports fewer than 2% violations.The analysis also finds a nonlinear relationship between epistemic uncertainty and entropy variance that the Dirichlet model cannot capture.Source: Main Result · p. 28, Empirical Results · p. 28, Figure 15 · p. 29

G.2.1. PRELIMINARIES

• Cross-entropies upper-bound their corresponding entropies, with equality when the modeled distribution matches the empirical distribution.The appendix states these bounds for joint entropy, feature entropy, and conditional entropy.Source: §G.2.1 · pp. 29–30

G.2.2. PROOF

• The conditional log-likelihood objective is strictly proper, while maximizing marginal log-likelihood makes the negative density-estimation objective lower-bounded by feature entropy, with equality at the empirical feature distribution.At the conditional optimum, the classifier captures the empirical conditional distribution.Source: §G.2.2 · p. 31
• For a GMM with one component per class, both objectives are minimized together when class features are perfectly separated, in which case GDA minimizes all objectives.If class features overlap, their optima do not coincide; randomly labeled samples drawn from the same Gaussian provide an example.Source: §G.2.2 · p. 31
• With noisy real-world data, a shared optimum is rare; otherwise, neither GDA nor another one-component-per-class GMM optimizes both objectives, while EM gives a better density estimator and softmax a better classifier.Gradient descent on either the softmax or density objective moves away from the GDA optimum.Source: §G.2.2 · p. 31
• An adapted Parzen estimator is a trivial optimal minimizer of the joint-entropy objective for small enough σ, showing that the proposition is not general.The stated construction is for an empirical data distribution.Source: §G.2.2 · p. 32

Intuitions & validation with a toy example

• The LDA solution, equivalent to a softmax linear layer, optimizes p(y | z) and uses overlapping Gaussian components to express aleatoric uncertainty, but its density fit is loose.Optimizing negative log-likelihood is a proper scoring rule for calibrated predictions.Source: §G.2.3 · pp. 32–33
• The feature-density objective does not depend on labels, so its components need not model classes and it does not provide class predictions.The paper therefore omits label-based scores and an entropy plot for this objective.Source: §G.2.3 · pp. 32–33
• The GDA’s tighter density comes at the cost of higher NLL and weaker aleatoric uncertainty; it can assign low entropy to an ambiguous point and be overconfident.Untied covariance matrices can also produce curved decision boundaries and unexpected changes far from the data.Source: §G.2.3 · pp. 32–33
• Each solution best minimizes its own objective: the GMM gives the best density fit, the LDA the best label NLL, and the GDA an almost-as-good density model.Table 12 reports the realized objective scores for the synthetic example; smaller is better.Source: Table 12 · p. 32, §G.2.3 · p. 32
• On Dirty-MNIST, softmax entropy separates ambiguous from unambiguous in-distribution samples but confounds in-distribution and OoD samples; feature density separates OoD only with ResNet-18+SN.Without those inductive biases, LeNet and VGG-16 feature collapse makes in-distribution and OoD densities overlap; density also confounds ambiguous and unambiguous in-distribution samples.Source: Figure 17