PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence Calibration for Convolutional Neural Networks Using Structured Dropout

Zhilu Zhang, Adrian V. Dalca, Mert R. Sabuncu

arXiv:1906.09551v1cs.LGcs.CVstat.ML

TL;DR

Neural networks can produce over-confident predictions, while uncertainty estimates are important for consequential decisions. This paper views MC dropout as an ensemble and uses structured dropout to promote diversity; experiments report improved calibration, with gains depending on the setting.

  • Problem

    Maximum-likelihood neural networks can produce over-confident predictions, and MC dropout can also remain over-confident despite improving on deterministic networks.

    Source: §1 · p. 1

  • Method

    The paper interprets MC dropout as an ensemble and uses structured dropout to promote diversity among sampled models and improve confidence calibration.

    Source: §5 · p. 8

  • Results

    Structured dropout promotes model diversity and improves calibration, with performance gains depending on architecture, task, dropout rate, and strategy.

    Source: §5 · p. 8

  • Takeaways & Limitations

    In Bayesian active learning, the ultimate accuracy increase was 21.1 ± 1.1% with MC dropout and 25.7±1.4% with MC dropChannel.

    Source: §4.3 · p. 8

    Increasing dropout can raise diversity but reduce individual-model accuracy, so the dropout rate is treated as a hyperparameter chosen using validation NLL.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dropout. Motivated by this, we explore the use of structured dropout to promote model diversity and improve confidence calibration. We use the SVHN, CIFAR-10 and CIFAR-100 datasets to empirically compare model diversity and confidence errors obtained using various dropout techniques. We also show the merit of structured dropout in a Bayesian active learning application.

1 Introduction

Reliable uncertainty estimates matter alongside predictive performance, but maximum-likelihood neural networks can be over-confident. The paper frames MC dropout as an ensemble and proposes structured dropout to improve model diversity and confidence calibration.

  • Reliable confidence estimates are important for decision-making applications such as medical diagnosis and autonomous driving, while maximum-likelihood neural networks often produce over-confident point estimates.Source: §1 · p. 1
  • Bayesian neural networks can provide well-calibrated probabilistic predictions, but computational challenges motivate efficient alternatives such as MC dropout, which estimates uncertainty using dropout at test time.Source: §1 · p. 1
  • The paper attributes MC dropout's poor calibration to limited diversity among sampled models and proposes structured dropout, which improves diversity and confidence estimates across SVHN, CIFAR-10, and CIFAR-100.It also reports advantages over deep ensembles and evaluates structured-dropout uncertainty in Bayesian active learning.Source: §1 · p. 1

2 Related Work

Prior work includes structured dropout methods developed to improve test-time accuracy, as well as Bayesian and non-Bayesian approaches to calibrated confidence. The paper examines structured dropout as a way to obtain better confidence estimates.

  • DropBlock, SpatialDrop, and Stochastic Depth Net drop structured parts of neural networks and were proposed to boost test-time accuracy; this paper applies them to confidence estimation.Source: §2 · p. 2
  • Other approximate Bayesian neural-network methods can require sophisticated implementations, be harder to scale, or perform sub-optimally; MCMC methods can be computationally expensive and hyperparameter-sensitive.Laplace approximation and SWA-Gaussian are also described as posterior-approximation approaches.Source: §2 · p. 2
  • Temperature scaling, deep ensembles, and ensembles of training snapshots are alternatives for calibrated confidence, with explicit and snapshot ensembles requiring multiple models or saved checkpoints, respectively.Source: §2 · p. 2

3 Uncertainty Estimates with Structured Dropouts

The section recasts MC dropout as an ensemble of sampled models and uses error–ambiguity and calibration decompositions to motivate structured dropout as a way to increase model diversity.

3.1 MC Dropout as Ensembles of Dropout Models

  • MC dropout averages predictions from dropout-sampled models, treating each sampled weight set as an ensemble member.The sampled weights are drawn independently layer-wise from a dropout-based approximate distribution.Source: §3.1 · p. 2
  • The error–ambiguity decomposition expresses ensemble MSE through average member MSE and ambiguity, which measures model diversity.It implies that an ensemble can be more accurate when its members are both accurate and diverse.Source: §3.1 · p. 3
  • ECE depends on individual-model accuracy and ensemble diversity, and greater diversity can reduce calibration error when average individual-model accuracy is held constant.The section interprets the variation in true class probabilities across ensemble confidence level-sets as a component of the ECE decomposition.Source: §3.1 · p. 3

3.2 Improved Confidence Calibration with Structured Dropout

  • Spatially correlated image features can leak information through regular dropout, yielding similar representations and limited diversity in sampled CNN models.The section contrasts this with deep ensembles, whose independently initialized and stochastically optimized networks can learn divergent representations.Source: §3.2 · p. 3
  • Structured dropout replaces element-wise masks with patch-, channel-, or layer-level masks to promote diversity among sampled models and improve confidence calibration.The methods are named dropBlock, dropChannel, and dropLayer, respectively.Source: §3.2 · p. 4
  • Increasing dropout rates may increase diversity at the cost of individual-model accuracy, so the rate is selected using validation NLL.Layer-level dropout also requires skip connections to preserve information flow when a layer is dropped.Source: §3.2 · p. 4

4 Experiments

The experiments compare structured dropout with regular MC dropout and deep ensembles across uncertainty calibration, model diversity, and CIFAR-10 active learning. MC dropLayer and MC dropChannel lead the calibration comparison, while structured dropout also shows larger accuracy gains as active-learning labels accrue.

4 Experiments

  • Across SVHN, CIFAR-10, and CIFAR-100, the study varied dropout type in PreAct-ResNets of 18, 50, and 152 layers; dropLayer sampled whole blocks at test time, except downsampling blocks.Downsampling blocks were dropped only during training.Source: §4 · pp. 4–5
  • Dropout variants used 30 test-time samples for Monte Carlo estimates; deep ensembles averaged five deterministic networks trained with random initializations.The dropout rate before the fully connected layer was 0.1 for all experiments, including deterministic models for a fair comparison.Source: §4 · p. 4

4.1 Evaluation of Uncertainty Estimates

  • MC dropLayer and MC dropChannel offered the best confidence calibration with good accuracy and lower ECE than a deep ensemble requiring 5 times more computational resource; regular MC dropout performed worse on every metric on both CIFAR datasets.Source: §4.1 · p. 5

4 Experiments

  • Adversarial training hampered both calibration and classification performance significantly, so the authors excluded it from training.This observation agrees with a recent finding cited in the paper.Source: §4 · p. 5

4.2 Ensemble Diversity

  • Structured dropout gained more from ensemble averaging; at five models, MC dropBlock beat deep ensembles on ECE, while IA showed greater diversity than regular dropout without one universally best method.The diversity comparison fixes dropout rates at 0.1; the authors suggest architecture and dataset may influence which structured method yields greater diversity.Source: §4.2 · pp. 6–7, Figure 2 · p. 7, Table 2 · p. 7

4.3 Bayesian Active Learning

  • 25.7±1.4% (MC dropChannel) versus 21.1 ± 1.1% (MC dropout) ultimate accuracy increase under Variation Ratios; structured methods gained more as labeled data accrued.The paper reports similar trends for the other acquisition functions and says Max Entropy and Variation Ratios seemed more effective in this experiment.Source: Figure 3 · p. 8, §4.3 · p. 8

5 Conclusion and Future Work

The paper interprets MC dropout as ensemble averaging and links calibration error to limited diversity among sampled models. Structured dropout can promote diversity and improve calibration, though gains depend on the setting.

5 Conclusion and Future Work

  • Structured dropout can promote model diversity and improve confidence calibration while remaining simple to implement and computationally efficient.The paper attributes MC dropout's calibration error to a lack of diversity among sampled models.Source: §5 · p. 8
  • Calibration gains depend on architecture, task, dropout rate, and the structured dropout strategy.Source: §5 · p. 8
  • Evaluation covered only ResNet among MC dropLayer architectures, used a constant dropout rate, and focused on CNN computer-vision tasks.The impact of varying or scheduling dropout rates remains open; the authors suggest extending the idea beyond CNNs.Source: §5 · p. 8

Appendix A: Performance of Uncertainty Estimates Against Dropout Rate

Figure 4 examines test-time NLL and accuracy as dropout rate changes across three datasets. Structured dropout can improve NLL at moderate rates, while DropLayer is comparatively rate-insensitive.

Appendix A: Performance of Uncertainty Estimates Against Dropout Rate

  • Structured dropout can achieve better NLL, particularly at moderate dropout rates, across SVHN, CIFAR-10, and CIFAR-100.Source: Figure 4 · p. 12
  • DropLayer is the least sensitive to dropout-rate choice with respect to NLL.Source: Figure 4 · p. 12
  • For dropBlock, NLL rises sharply after its minimum on all three datasets, possibly because blocks are too large in later layers with similarly sized feature maps.Source: Figure 4 · p. 12

Appendix B: Additional Results with Active Learning

Figure 5 reports CIFAR-10 active-learning accuracy for dropout methods using Max Entropy or BALD. MC dropout yields the least relative accuracy improvement, as also observed with Variation Ratios.

Appendix B: Additional Results with Active Learning

  • MC dropout yields the least relative test-accuracy improvement across methods, similar to results with Variation Ratios.Source: Figure 5 · p. 13
  • The figure plots test accuracy against training-sample count for Max Entropy and BALD acquisition on CIFAR-10.Source: Figure 5 · p. 13
  • The right panel reports relative test-accuracy improvement over the first iteration for each dropout method.Source: Figure 5 · p. 13
Loading 1906.09551v1…