Robustly representing uncertainty in deep neural networks through sampling

1 Introduction

• Bayesian DNNs target epistemic uncertainty over parameters, which softmax outputs do not capture, but computing the ideal posterior is intractable for large networks.Softmax outputs represent aleatoric uncertainty in the training data.Source: §1 · p. 1
• Multiplicative Bernoulli dropconnect and Gaussian multiplicative weight sampling had not been thoroughly evaluated, despite prior work on dropout-based Bayesian CNNs.Gal and Ghahramani and Kingma et al. [13] developed Bayesian CNNs using Bernoulli and Gaussian dropout, respectively.Source: §1 · p. 1
• Some variational methods substantially increase network parameters, while an earlier spike-and-slab implementation more than doubled learned parameters and did not consistently outperform standard networks.That approach learned each weight's mean and standard deviation, as well as each unit's dropout rate.Source: §1 · pp. 1–2
• The authors evaluate Bayesian CNNs on MNIST and CIFAR-10, test uncertainty under added image noise, and propose combining Bernoulli dropout with Gaussian dropconnect without adding learned parameters.They present the combination as an approximation to spike-and-slab variational inference, intended to combine robust uncertainty estimates with test-set generalization.Source: §1 · p. 2

2.1 Bayesian Deep Neural Networks

• Variational inference optimizes parameters V so qV(W) approximates the posterior p(W|Dtrain), with Monte Carlo weight samples estimating the likelihood objective and test-data probabilities.The objective is to minimize the Kullback–Leibler divergence between qV(W) and the posterior.Source: §2.1 · p. 2

2.2 Variational Distributions

• These variational methods sample weights by element-wise multiplication of variational parameters by a random mask, with the mask distribution distinguishing dropout and dropconnect variants.Figure 1 illustrates sampling network weights under the different variational distributions.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.2.1 Bernoulli Dropconnect & Dropout

• Bernoulli dropconnect independently samples a mask for each connection, whereas Bernoulli dropout samples one mask value per weight-matrix row, corresponding to dropping a unit.The drop probability p applies to connections in dropconnect and units in dropout.Source: §2.2.1 · p. 3

2.2.2 Gaussian Dropconnect & Dropout

• Gaussian dropconnect independently samples mask values for connections, while Gaussian dropout shares a random value across each row; both use masks drawn from Gaussian distributions.The Gaussian is centered at the variational parameter for sampled weights; the stated dropconnect variance matches Bernoulli dropout when training-time scaling is used.Source: §2.2.2 · p. 3

2.2.3 Spike-and-Slab Dropout

• The proposed spike-and-slab approximation combines Bernoulli dropout with Gaussian dropconnect, using row-level Bernoulli masking and independent Gaussian weight masks without adding learned parameters.At non-sampling inference, it uses mean weights and scales unit outputs by 1 − pdo.Source: §2.2.3 · p. 3

3.1 Logistic Regression

• In the logistic-regression visualization, all MC methods showed greater uncertainty farther from training data, especially dropconnect and spike-and-slab methods.Decision-boundary variability represents uncertainty in this visualization.Source: §3.1 · p. 4

3.2 Convolutional Neural Networks

• CNNs trained on MNIST and CIFAR-10 were tested on original and progressively noise-corrupted images to assess uncertainty in input regions not seen during training.Source: §3.2 · pp. 4–5
• With added image noise, sampling consistently improved calibration; Gaussian dropout had the highest test accuracy but less noise robustness than Bernoulli and spike-and-slab dropout.Source: §3.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5
• On CIFAR-10, MC sampling outperformed standard dropout at higher dropout variance and made performance less dependent on the dropout probability.For spike-and-slab dropout, the comparison varied pdo while holding pdc fixed.Source: §3.2 · p. 6, Figure 6 · p. 6

• Dropconnect-based CNNs performed worse on the unmodified test set but were more robust to deviations from the training distribution.Source: §4 · p. 7
• Dropout-based networks, particularly MC Gaussian dropout, performed well on the unmodified test set but were less robust to distribution deviations.Source: §4 · p. 7
• Concurrent Bernoulli dropout and Gaussian dropconnect approximated spike-and-slab sampling, performing better near the test set than dropconnect methods and representing uncertainty more robustly than dropout methods.Source: §4 · p. 7

• The supplementary-material heading identifies uncertainty through sampling as its topic.Source: Supplementary Material: Robustly… · p. 9
• Patrick McClure is listed with the MRC Cognition and Brain Sciences Unit.Source: Supplementary Material: Robustly… · p. 9

1.1 L2 regularization and the KLD between Gaussians

• For a zero-mean prior, the fixed-variance Gaussian KL is equivalent to L2 regularization; when the variational variance depends on parameters, the paper instead minimizes a lower bound.The derivation uses that a sum of variance-dependent terms and a constant is nonnegative; a similar lower bound applies to multivariate Gaussians.Source: §1.1 · p. 9

1.2 Gaussian "reparameterization trick"

• The reparameterization trick expresses Gaussian weight sampling through standard-normal noise and a deterministic, differentiable, monotonic mapping.The noise variables are standard normal, and α = p/(1 − p), where p is the dropout or dropconnect probability.Source: §1.2 · pp. 9–10

1.3 MC Gaussian Dropconnect and Dropout

• Gaussian dropout and dropconnect learn a variational weight distribution, then approximate training and test expectations with Monte Carlo integration.Dropconnect samples noise per weight element; dropout samples per unit.Source: §1.3 · p. 10

1.4 MC spike-and-slab Dropout

• Spike-and-slab dropout models weights as Bernoulli masks multiplied by Gaussian weights, with independent variational factors and an L2-based lower bound on the Gaussian KL.The method uses Monte Carlo integration for training and testing; the Bernoulli prior yields zero KL for the mask distribution.Source: §1.4 · pp. 10–11

2.1 Architectures

• The architecture tables specify the CNNs used for MNIST and CIFAR-10.The supplied captions identify the dataset for each architecture table but do not provide architecture details.Source: Table 1 · p. 12, Table 2 · p. 12

2.2 Additional results

• Figure 1 plots CIFAR-10 classification error and calibration MSE across Gaussian image-noise levels for five masking methods, with and without 10-sample MC prediction.It also plots calibration MSE against classification error; the captions give p = 0.5 for dropout and dropconnect, and pdo = 0.5 and pdc = 0.1 for spike-and-slab.Source: Figure 1 · p. 13
• Figure 2 plots true-label frequency against predicted probability for CIFAR-10-trained BDC, GDC, BDO, GDO, and SSD across Gaussian image-noise standard deviations, with and without 10-sample MC sampling.The x = y line is ideal; p = 0.5 for dropconnect and dropout, while SSD uses pdo = 0.5 and pdc = 0.1.Source: Figure 2 · p. 13