DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks

Aryan Mobiny; Hien V. Nguyen; Supratik Moulik; Naveen Garg; Carol C. Wu

Source-linked AI summary

DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks

Aryan Mobiny, Hien V. Nguyen, Supratik Moulik, Naveen Garg, Carol C. Wu

arXiv:1906.04569v1Jun 7, 2019cs.LGcs.AIcs.CVstat.ML

TL;DR

Safety-critical DNN applications need uncertainty estimates because point predictions and softmax outputs do not reliably indicate model confidence, while traditional Bayesian inference is computationally difficult. The paper introduces MC-DropConnect, which applies Bernoulli distributions directly to network weights to approximate Bayesian inference. Across classification and semantic segmentation experiments, it reports improved prediction accuracy and uncertainty estimation, with uncertainty providing additional information for decision-making.

Problem
Softmax outputs and point-estimate training do not provide reliable model uncertainty, while traditional Bayesian neural networks are computationally expensive for large architectures.
Method
MC-DropConnect approximates variational Bayesian inference by imposing Bernoulli distributions directly on neural-network weights without adding model parameters or increasing time complexity.
Results
MC-DropConnect improves prediction accuracy and uncertainty estimation across evaluated classification and semantic segmentation settings, with erroneous predictions receiving significantly higher uncertainty values (Kolmogorov-Smirnov test, p-value < 0.001).
Takeaways & Limitations
The uncertainty estimates provide additive information that can assist users and automated systems in decision-making and prediction rejection.

Abstract

from arXiv · show

Deep neural networks (DNNs) have achieved state-of-the-art performances in many important domains, including medical diagnosis, security, and autonomous driving. In these domains where safety is highly critical, an erroneous decision can result in serious consequences. While a perfect prediction accuracy is not always achievable, recent work on Bayesian deep networks shows that it is possible to know when DNNs are more likely to make mistakes. Knowing what DNNs do not know is desirable to increase the safety of deep learning technology in sensitive applications. Bayesian neural networks attempt to address this challenge. However, traditional approaches are computationally intractable and do not scale well to large, complex neural network architectures. In this paper, we develop a theoretical framework to approximate Bayesian inference for DNNs by imposing a Bernoulli distribution on the model weights. This method, called MC-DropConnect, gives us a tool to represent the model uncertainty with little change in the overall model structure or computational cost. We extensively validate the proposed algorithm on multiple network architectures and datasets for classification and semantic segmentation tasks. We also propose new metrics to quantify the uncertainty estimates. This enables an objective comparison between MC-DropConnect and prior approaches. Our empirical results demonstrate that the proposed framework yields significant improvement in both prediction accuracy and uncertainty estimation quality compared to the state of the art.

I. INTRODUCTION

Deep neural networks lack reliable model uncertainty despite their use in safety-critical applications, while Bayesian approaches are often computationally expensive. The paper introduces MC-DropConnect, a tractable Bernoulli-weight approximation to Bayesian inference, and evaluates it across classification and semantic segmentation.

Motivation: Standard ML and MAP training produces point estimates, while softmax probabilities do not represent overall model confidence.A model may remain uncertain even with a high softmax output.
Motivation: Reliable uncertainty estimates can help users recognize under-confident or falsely over-confident predictions in safety-critical automated decisions.The motivation includes applications such as automated cancer detection.
Background: Bayesian neural networks model uncertainty through weight priors and output distributions, but large parameter counts make them computationally expensive.Gaussian variational approximations further increase the required number of parameters.
Proposed Method: MC-DropConnect imposes a Bernoulli distribution directly on network weights to approximate the posterior over weight matrices.The paper derives equations for this mathematically grounded approximation of variational inference in Bayesian neural networks.
Contributions: The method uses existing tools without additional model parameters or increased time complexity, and is evaluated on image classification and semantic segmentation.The paper also proposes metrics for evaluating uncertainty estimation and reports improved prediction accuracy and robust generalization compared with standard techniques.
Related Work: Prior alternatives face computational costs from posterior sampling, repeated forward passes, or maintaining several separate models.These limitations are described for Hamiltonian Monte Carlo, MC-Dropout, and Deep Ensembles.

III. METHODOLOGY

The methodology combines Bayesian neural networks, variational inference, and DropConnect to approximate Bayesian modeling and evaluate predictive uncertainty.

The method applies Bernoulli distributions to all network weights and uses uncertainty metrics to evaluate the resulting Bayesian approximation.

A. Bayesian Neural Networks

Bayesian neural networks place distributions over weights to produce predictive uncertainty, but exact inference is computationally intractable. Variational inference addresses this by optimizing a tractable approximation to the posterior.

Bayesian neural networks place a prior distribution over weights, providing model regularization and uncertainty estimates absent from maximum-likelihood training.
Bayesian inference computes the posterior over weights and integrates predictions across that posterior for an unknown test label.
Exact predictive inference is equivalent to averaging over an uncountably infinite ensemble of weight configurations, making it computationally intractable.
Variational inference introduces qθ(w) and minimizes its KL divergence from the true posterior.
The negative ELBO objective balances explaining observed data through expected log likelihood against prior-driven regularization through the prior KL term.
For semantic segmentation, the predictive and uncertainty computations apply pixel-wise, analogously to classification.

B. DropConnect

DropConnect generalizes Dropout by applying Bernoulli dropping to individual weights rather than output units, producing stochastic network activations for regularization.

A standard fully connected layer computes activations as ai = σ(Wiv) from an input vector and weight matrix.
Dropout multiplies layer outputs by a binary vector whose elements are sampled from Bernoulli distributions with keep probabilities pi.
DropConnect applies Bernoulli dropping directly to each weight, using a binary matrix Zi with the same shape as Wi.

C. DropConnect for Bayesian Neural Network Approximation

The method approximates Bayesian inference by modeling each network weight with a Bernoulli distribution and training with DropConnect. At test time, stochastic forward passes provide Monte Carlo samples from the approximate posterior.

Variational approximation: Each layer approximates its weight posterior with a Bernoulli-distributed binary mask and optimized variational parameters.The mask entries determine whether individual weights are retained or dropped.
Training: The intractable Bayesian objective is estimated by reparameterizing weights through Bernoulli sampling and using a single Monte Carlo sample during training.The sampled weights are realizations from qθ(w), not maximum a posteriori estimates.
Training: The resulting objective matches standard neural-network training with L2 regularization and DropConnect applied to all weights.This provides an unbiased scaled estimator of the Bayesian objective while retaining the regularization effect.
Inference: Inference replaces the posterior with the approximate distribution and estimates the predictive integral using Monte Carlo integration.Multiple stochastic forward passes approximate the posterior distribution of softmax class probabilities.
Inference: DropConnect remains active at test time, so each forward pass samples network weights and the mean of these samples becomes the prediction.The paper calls this approach MC-DropConnect.

D. Measuring the Model Uncertainty

The paper distinguishes uncertainty arising from limited model knowledge from irreducible data noise. It estimates model uncertainty from stochastic predictions, emphasizing mutual information as a targeted measure.

Types of uncertainty: Epistemic uncertainty reflects insufficient training data and can be reduced with enough additional data, whereas aleatoric uncertainty represents irreducible data noise.Examples of aleatoric noise include motion and sensor noise.
Uncertainty measures: Posterior-sample variance provides a direct uncertainty estimate for each class, while predictive entropy captures both epistemic and aleatoric uncertainty.The paper notes that predictive entropy is not ideal when the focus is specifically model uncertainty.
Uncertainty measures: Mutual information measures the information gained about model parameters from receiving a test input and its true label.It is informative in regions where the model is uncertain and less informative where the model is well established.
Uncertainty measures: The paper approximates mutual information with Monte Carlo softmax predictions, whose class probabilities are averaged across T stochastic samples.The resulting quantity can be computed for each DropConnect Monte Carlo run.

E. Unceratinty Evaluation Metrics

The proposed evaluation framework converts uncertainty assessment into a binary prediction problem using correctness and confidence maps. It summarizes performance across uncertainty thresholds with area under each metric curve.

Evaluation framework: Uncertainty evaluation is difficult because uncertainty values lack ground truth and therefore have no clear direct definition of quality.The paper presents metrics to address this evaluation challenge.
Evaluation framework: The metrics combine ground-truth labels, model predictions, and uncertainty values and apply to classification and semantic segmentation.Semantic segmentation is treated as pixel-wise classification for this purpose.
Evaluation framework: The procedure maps correct and incorrect predictions, thresholds uncertainty into certain and uncertain cases, and treats the results as a binary prediction task.The uncertainty threshold varies between the minimum and maximum uncertainty over the test set.
Metrics: Negative predictive value measures whether certain predictions are correct, while true positive rate measures whether incorrect predictions receive high uncertainty.High epistemic uncertainty can flag wrong predictions for further precautions.
Caveat: A correct prediction need not be certain, because unfamiliar poses or conditions can produce correct but relatively uncertain detections.Thus, the converse of the desired uncertainty relationships does not necessarily hold.
Metrics: Metric values are compared by area under curves plotted against normalized uncertainty thresholds.Higher values indicate better performance for the proposed metrics.

IV. EXPERIMENTAL RESULTS AND DISCUSSION

The experiments assess DropConnect-based uncertainty estimates on classification and semantic segmentation, comparing MC-DropConnect with MC-Dropout across multiple datasets. The paper reports considerable improvements in prediction accuracy and uncertainty estimation quality.

Experimental scope: Experiments evaluate DropConnect CNN uncertainty estimates on classification and semantic segmentation tasks.The evaluation uses a range of datasets and compares the proposed method with MC-Dropout.
Results: The paper reports considerable improvement in prediction accuracy and uncertainty estimation quality relative to the state-of-the-art comparison method.The uncertainty estimates are quantitatively evaluated with the proposed metrics.

A. Classification

In classification experiments, MC-DropConnect achieved comparable or marginally improved prediction accuracy while substantially improving uncertainty estimation. Its uncertainty estimates also supported detecting likely errors, handling domain shift, referring uncertain samples, and converging with fewer stochastic passes than MC-Dropout.

Classification results: Across varying uncertainty thresholds, MC-DropConnect generally outperformed MC-Dropout on the proposed uncertainty metrics for MNIST and CIFAR-10.The comparison is shown in Fig. 2, with MC-DropConnect in green and MC-Dropout in red.
Classification results: MC-DropConnect produced marginally improved prediction accuracy with MC sampling and significantly improved uncertainty-estimation metrics on MNIST and CIFAR-10.The experiments used LeNet and FCNet Bayesian estimates, with comparisons summarized in Table I and Fig. 2.
Uncertainty-informed referrals: High-uncertainty sample rejection increased CIFAR-10 prediction accuracy, and comparison with random rejection indicated that the uncertainty contained information about prediction performance.The referral experiment sorted predictions by epistemic uncertainty measured with mutual information and evaluated accuracy over tolerated uncertainty and retained-data fractions.
Convergence: MC-DropConnect reached lower test error than the baseline after 2 samples and came within one standard deviation of its best performance after 18 samples, versus 6 and 54 samples for MC-Dropout.The best MC-DropConnect performance occurred at T=90, while MC-Dropout reached its best performance at T=94.

B. Semantic Segmentation

The semantic-segmentation experiments evaluate MC-DropConnect against MC-Dropout across road-scene and medical datasets. MC-DropConnect generally improves segmentation accuracy and provides uncertainty estimates that track prediction correctness and confidence.

Experimental setup: The experiments compare MC-DropConnect with MC-Dropout on CamVid, CityScapes, and 3D CT-Organ segmentation tasks.The study uses SegNet for CamVid, ENet for CityScapes, and V-Net for volumetric CT-organ segmentation.
Quantitative observations: MC-DropConnect generally outperforms MC-Dropout in quantitative uncertainty estimation as the uncertainty threshold varies.The comparison is shown for CamVid, CityScapes, and CT-Organs.
Qualitative observations: The uncertainty maps from MC-DropConnect display higher model uncertainty when the models make wrong predictions.The qualitative comparison uses correctness and confidence maps for both methods.
Qualitative observations: Higher uncertainty is associated with object boundaries, rare classes, and visually ambiguous class pairs.Examples include pole and sign symbols in CamVid, bicyclist versus pedestrian, and car versus truck.
Quantitative observations: MC-DropConnect generally improves the accuracy of predicted segmentation masks for all three model-dataset pairs.The comparison is reported using pixel accuracy, mean accuracy, and mean IOU metrics.
Confidence analysis: Very high accuracy at the 90th-percentile confidence level indicates that the uncertainty estimate effectively measures prediction confidence.The reported confidence analysis covers all three dataset-model pairs.

V. CONCLUSION

The conclusion presents MC-DropConnect as a mathematically grounded, computationally tractable approach for estimating model uncertainty in Bayesian neural networks. Experiments report improved prediction accuracy and precise confidence estimates, while future work targets the effect of DropConnect on learned kernels and uncertainty during training.

Conclusion: MC-DropConnect provides computationally tractable approximate inference in Bayesian neural networks without additional computational cost.The framework extracts information from existing models and represents model uncertainty.
Conclusion: The proposed metrics evaluate uncertainty-estimation performance across machine-learning tasks including regression, classification, and semantic segmentation.The metrics incorporate ground-truth labels, model predictions, and uncertainty values.
Conclusion: MC-DropConnect improves prediction accuracy and yields precise estimates of model confidence in its predictions.The conclusion reports this outcome empirically.
Conclusion: The uncertainty estimates provide additive information that can assist users in decision-making.This consequence is stated within the paper’s empirical conclusion.
Future work: Future work will study how DropConnect and different drop probabilities affect trained convolutional kernels and whether uncertainty can enrich knowledge of the data domain during training.These are proposed research directions rather than established results.