Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles

1 INTRODUCTION

• Specialized domains make expert-labeled data costly and time-consuming to acquire, while active learning aims to reduce the labels needed for a task.Active learning iteratively adds unlabeled examples to the training set using an acquisition function that ranks examples by importance.Source: §1 · p. 1
• Deep active learning faces both scalability challenges in estimating uncertainty and the need for reliable uncertainty estimates to avoid overconfident predictions.Batch sampling helps keep computational costs low; DBAL uses MC-dropout as a Bayesian framework for uncertainty estimates.Source: §1 · p. 1
• The authors hypothesize that mode collapse in variational inference contributes to overconfident DBAL predictions and propose combining ensembles with MC-dropout to improve uncertainty without trading representativeness.The paper contrasts uncertainty-based selection with density-based representativeness and hybrid methods.Source: §1 · p. 2
• The paper reports imbalanced training sets acquired by MC-dropout as evidence of mode collapse and presents DEBAL for experiments on MNIST and CIFAR-10.It also reports that preferential selection is not beneficial for the active-learning process.Source: §1 · p. 2

2 BACKGROUND

• There is no universal active-learning strategy, so researchers use heuristics tailored to particular tasks, as Dasgupta (2005) proved.Settles (2012) provides a comprehensive review of the area.Source: §2 · p. 2
• DBAL uses MC-dropout to approximate a posterior over network predictions and feed uncertainty estimates into acquisition, but variational approximations can suffer from mode collapse.The background also describes deterministic ensembles as an alternative uncertainty approximation and notes their reported performance and robustness.Source: §2 · p. 2
• Core-set selection chooses a diverse subset that represents the input space, but requires computationally expensive mixed-integer optimization.Ducoffe & Precioso (2018) instead use distances to adversarial examples to approximate distances to decision boundaries.Source: §2 · p. 2
• Hybrid active-learning methods combine uncertainty and representativeness, and Wang & Ye (2015) was, to the authors' knowledge, the only such method tested in CNN-based DAL.That method was reported to perform worse than the core-set approach in Sener & Savarese (2017).Source: §2 · p. 3
• MC methods approximate posterior expectations using finite samples, and MC-dropout replaces the posterior over parameters with a dropout distribution that scales to high-dimensional data.The paper describes MC-dropout as flexible and applicable to existing neural-network architectures.Source: §2 · p. 3
• DBAL distinguishes epistemic uncertainty from insufficiently learned parameters and aleatoric uncertainty from genuine stochasticity in the data.The background introduces entropy, BALD, and variation ratio as acquisition-function approaches to uncertainty.Source: §2 · p. 3

3.1 EXPERIMENTAL DETAILS

• Experiments use MNIST and CIFAR-10, retraining on the accumulated labeled set at each acquisition step and averaging results over three repetitions.Ensembles contain three networks with identical architectures and different random initializations; stochastic acquisition uses 100 MC-Dropout forward passes.Source: §3.1 · p. 4

3.2 EVIDENCE OF MODE COLLAPSE IN DBAL

• MC-Dropout acquisition yields a training set with limited class diversity, especially early in acquisition, and overrepresented classes are neither unusually uncertain nor easier to classify.The authors also report overconfident predictions in unfamiliar regions of the MNIST latent space, supporting their account of mode collapse.Source: §3.2 · pp. 4–5, Figure 1 · p. 4

3.3 DEEP ENSEMBLES: A RECIPE FOR USEFUL UNCERTAINTY ESTIMATION

• DEBAL averages predictions from independently trained MC-Dropout ensemble members, each initialized differently, and uses the resulting uncertainty for acquisition.The ensemble is treated as a uniformly weighted mixture; its stochastic form further averages predictions across MC-Dropout forward passes.Source: §3.3 · pp. 5–6
• DEBAL significantly improves classification accuracy on both MNIST and CIFAR-10 compared with a single MC-Dropout network.Figure 3 illustrates the comparison for BALD; the authors report similar results for other acquisition functions.Source: §3.3 · p. 6, Figure 3 · p. 7

3.4 DETERMINISTIC VS STOCHASTIC ENSEMBLE

• On seen MNIST and unseen NotMNIST, stochastic ensembles are more confident overall than deterministic ensembles, while their uncertain bins show lower classification accuracy.For MNIST, the authors report that accuracy decreases as uncertainty rises for both methods; the stochastic ensemble's additional confidence appears to reflect meaningful uncertainty.Source: §3.4 · p. 7, Figure 5 · p. 8
• The stochastic ensemble has a lower MNIST Brier score than the deterministic ensemble, 0.0244 versus 0.0297, indicating better uncertainty calibration.The authors also report that adding MC-Dropout stochasticity to a deterministic ensemble increases overall performance.Source: §3.4 · p. 8, Figure 6 · p. 8

• The study focuses on active learning within deep learning for image classification.Source: §4 · p. 8
• Mode collapse empirically harms the current state-of-the-art Bayesian active learning method.Source: §4 · p. 8
• Model ensembles improve on that method by leveraging their expressive power and statistical properties.Source: §4 · p. 8
• The paper links improved performance to better representation of data uncertainty and proposes exploring this for adversarial examples in safety-critical applications.Source: §4 · p. 8

• Figure 7 shows histograms of the 1,000 most uncertain MNIST test samples ranked by a LeNet model trained on the full training set.Source: Figure 7 · p. 12
• Figure 8 presents end-of-process MNIST confusion matrices on the 10,000-image test set, with the fully trained model as a baseline.Source: Figure 8 · p. 13
• Figure 9 compares the training-set true-label histogram after eight acquisition iterations for a single MC-dropout network and a three-network ensemble.The training set contains 100 images.Source: Figure 9 · p. 14
• Figure 10 visualizes end-of-process MNIST uncertainty, with lighter regions indicating higher uncertainty, for an ensemble of three similar networks and a single network.Source: Figure 10 · p. 14
• Figure 11 shows MNIST tSNE embeddings comparing ensembles of three similar stochastic or deterministic models with one MC-dropout network; orange points mark acquired images.Source: Figure 11 · p. 15