Source-linked AI summary
BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
Andreas Kirsch, Joost van Amersfoort, Yarin Gal
TL;DR
Deep Bayesian active learning needs data-efficient batch selection, but selecting points independently can create redundant batches. BatchBALD approximates batch–parameter mutual information with a greedy, tractable procedure, and the paper reports more diverse acquisitions, improved performance, and fewer labels and retraining steps, with limitations on unbalanced test sets and unlabeled-data use.
Problem
Batch acquisition in active learning must reduce labeling and retraining costs without selecting jointly redundant points.
Method
BatchBALD approximates mutual information between batches and model parameters using a greedy algorithm with a 1 − 1/e guarantee and efficient computation.
Results
BatchBALD produces more diverse acquisitions and improves performance over BALD and other methods while reducing the number of labeled points and model retraining steps.
Takeaways & Limitations
Accounting for dependencies within acquisition batches can improve practical data efficiency in deep Bayesian active learning.
Takeaways & Limitations
BatchBALD does not use information from the unlabeled dataset and is not expected to work well when the test set is unbalanced.
Abstract
from arXiv · showhide
We develop BatchBALD, a tractable approximation to the mutual information between a batch of points and model parameters, which we use as an acquisition function to select multiple informative points jointly for the task of deep Bayesian active learning. BatchBALD is a greedy linear-time $1 - \frac{1}{e}$-approximate algorithm amenable to dynamic programming and efficient caching. We compare BatchBALD to the commonly used approach for batch data acquisition and find that the current approach acquires similar and redundant points, sometimes performing worse than randomly acquiring data. We finish by showing that, using BatchBALD to consider dependencies within an acquisition batch, we achieve new state of the art performance on standard benchmarks, providing substantial data efficiency improvements in batch acquisition.
1 Introduction
Deep learning’s data hunger motivates active learning, but batch acquisition must select points that are informative jointly rather than merely individually. BatchBALD addresses this redundancy problem with an efficient greedy batch-selection method.
- Motivation: Active learning seeks to minimize labeling while reaching a suitable accuracy by iteratively querying labels for informative unlabeled points.The approach reduces repeated collection and labeling of large datasets.
- Motivation: Batch acquisition reduces model retraining and expert-time costs, which are especially significant for large models and expensive labeling tasks.Retraining can become a computational bottleneck, while expert labeling may involve costly medical specialists.
- Problem: In datasets with replicated points, BALD can perform worse than random acquisition, whereas BatchBALD handles replication and outperforms BALD.The repeated-MNIST setting exposes the data-efficiency failure caused by selecting replicas.
- Problem: Naive batch BALD selects the top b individually informative points, which can produce jointly redundant acquisitions.The method may select near-duplicates instead of covering multiple informative regions.
- Contributions: A greedy batch-selection algorithm runs in linear time and achieves a worst-case 1 − 1/e approximation to the optimal acquisition for the proposed function.This avoids enumerating exponentially many candidate subsets.
- Contributions: BatchBALD is presented as a data-efficient method for acquiring image batches, with improved data efficiency and reduced total runtime.Its contributions include a method for high-dimensional image data and supporting algorithmic components.
2 Background
Bayesian active learning scores unlabeled points using model uncertainty and mutual information, while practical batch acquisition creates a need to account for joint informativeness. The paper uses Bayesian neural networks and develops BatchBALD-oriented approximations for this setting.
- Bayesian active learning: The Bayesian active learning setup seeks a target prediction accuracy with the fewest oracle queries from an unlabeled pool.It maintains a training set, a Bayesian model posterior, and predictive distributions for candidate points.
- Bayesian active learning: An acquisition function scores candidate batches of unlabeled points using the current posterior over model parameters.The candidate batch consists of multiple points selected from the pool.
- BALD: BALD estimates mutual information between model predictions and parameters, prioritizing labels expected to inform the true parameter distribution.High mutual information corresponds to uncertain predictions that become more consistent across posterior parameter draws.
- BALD: BALD was designed for individual acquisitions, but batch use commonly selects the top b individual scores while retraining remains costly.This approach does not directly optimize joint informativeness.
- Bayesian neural networks: The paper uses Bayesian neural networks with variational inference approximated by MC dropout for scalable deep-learning applications.MC dropout is chosen for implementation simplicity and scalability to large models and datasets.
3 Methods
BatchBALD jointly scores candidate points by their mutual information with model parameters, accounting for overlaps that make independent BALD selection redundant. A greedy submodular approximation and Monte Carlo computation make batch acquisition tractable at larger scales.
- 3.1 BatchBALD: BatchBALD estimates the mutual information between a joint of multiple data points and the model parameters.
- 3.1 BatchBALD: Independent BALD selection can double-count overlapping information, whereas BatchBALD seeks a more diverse cover of model uncertainty.
- 3.1 BatchBALD: For acquisition size 1, BatchBALD and BALD are equivalent, while generally aBatchBALD ≤ aBALD.
- 3.2 Greedy approximation algorithm for BatchBALD: The BatchBALD acquisition function is submodular, so greedy selection achieves a 1 − 1/e approximation to the optimal batch.
- 3.3 Computing aBatchBALD: Conditional independence makes the conditional joint entropy decompose into a sum, while the unconditioned joint probability requires Monte Carlo estimation over model-parameter samples.
- 3.3 Computing aBatchBALD: Monte Carlo sampling bounds full BatchBALD complexity at O(bc · min{c^b, m} · |Dpool| · k), versus O(c^b · |Dpool|^b · k) for exact optimal batches.
4 Experiments
Across repeated MNIST, MNIST, EMNIST, and CINIC-10 experiments, BatchBALD improves batch acquisition performance, preserves diversity, and can reduce data or training time relative to BALD and random acquisition.
- Repeated MNIST: On repeated MNIST with near-duplicate points, BALD performs poorly and can underperform random acquisition, whereas BatchBALD handles replication effectively.Variation Ratios and Mean STD perform on par with random acquisition in this setting.
- MNIST: BALD performance drops substantially as acquisition size increases from 1 to 40, while BatchBALD maintains performance from size 5 to 10 and drops only slightly at 40.The slight drop at acquisition size 40 may be due to estimator noise.
- MNIST: With acquisition size 10, BatchBALD outperforms BALD and performs close to the ideal acquisition size of 1.BatchBALD with size 10 is much faster than BALD with size 1 and only marginally slower than BALD with size 10 when reaching 95% accuracy.
- EMNIST: On EMNIST with acquisition size 5, BatchBALD outperforms random acquisition and BALD, while BALD does not beat random acquisition.BatchBALD also acquires a more diverse set of data points, reflected by consistently higher entropy of acquired class labels and more consistent class sampling.
- CINIC-10: On CINIC-10 transfer learning, BatchBALD reaches 59% accuracy after 1170 data points versus 1330 for BALD at the median.The experiment uses an ImageNet-pretrained VGG-16 model and acquisition size 10.
5 Related work
Related work addresses parallel acquisition, batch diversity, and combinations of active learning with semi-supervised learning. These approaches include Gaussian-process-based Bayesian optimisation, constrained or mixture-model methods, and hybrid AL methods.
- Bayesian optimisation: Bayesian optimisation is related to active learning through its goal of finding an optimum with few function evaluations, but parallel evaluation remains an open problem.The related discussion notes that Bayesian optimisation commonly uses a Gaussian Process and often has a single worker.
- Diverse acquisition: Batch diversity has been pursued with constrained optimisation and Gaussian Mixture Models, while noisy BALD estimates have been proposed for molecular-data acquisition.The molecular-data approach limits the number of MC dropout samples to address BALD’s lack of diversity.
- Semi-supervised learning: Semi-supervised learning differs from active learning by commonly treating labelled data as fixed and using unlabelled data for unsupervised learning; prior work explores combining the two.The cited combination work includes approaches by Wang et al., Sener and Savarese, and Samarth Sinha.
6 Scope and limitations
BatchBALD has scope boundaries involving class balance, unlabeled-data information, and estimator noise. These constraints affect where it is expected to perform well and where larger batches may benefit from improved estimation.
- BatchBALD and BALD do not work well when the test set is unbalanced because they aim to learn about all classes rather than follow dataset density.
- BatchBALD does not use information from the unlabeled dataset, although semi-supervised learning could improve uncertainty estimates by modeling feature-space structure.
- Noise from MC-dropout’s variational approximation and joint-entropy sampling may reduce the quality of larger acquisition batches.
7 Conclusion
The paper introduces BatchBALD and a greedy algorithm for selecting informative batch candidates. Its acquisitions are more diverse and improve performance, while reducing labeling and retraining demands despite added acquisition-time computation.
- BatchBALD introduces a batch acquisition function and a greedy algorithm that selects good candidate batches relative to the intractable optimum.
- BatchBALD can significantly reduce labeled data and model retraining rounds, potentially saving considerable costs despite additional acquisition computation.
A Proof of submodularity
The proof establishes that the BatchBALD acquisition function is submodular, meaning additional points provide diminishing returns. This property yields a greedy 1 − 1/e approximation to the optimal batch.
- Submodularity expresses diminishing returns when additional points are added to the acquisition function.
- BatchBALD is shown to be submodular over subsets of the pool.
- The proof derives submodularity by showing that separate conditional mutual informations are at least as large as their joint conditional mutual information.
- The argument uses entropy subadditivity and additivity under conditional independence of the labels given model parameters.
- 1 − 1/e is the resulting approximation guarantee for the greedy algorithm.
- BALD approximates BatchBALD and provides an upper bound on the corresponding BatchBALD score.
B.2 BatchBALD as an approximation of BALD with acquisition size 1
The greedy BatchBALD inner loop conditions each candidate on points already selected, linking it to single-point BALD after retraining. The implementation caches factored probability terms and uses matrix multiplication for efficient computation.
- Each greedy step maximizes a candidate’s mutual information with model parameters conditioned on previously selected points and the existing training set.
- Under the stated comparison, each inner-loop step is no better than retraining on newly labeled points and selecting the next point with single-point BALD.
- Across the active training loop, the method computes a greedy 1 − 1/e approximation to the mutual information of all acquired points with the model parameters.
- The implementation factors the joint predictive probability and uses importance sampling to approximate the entropy calculation.
- Caching the previous probability matrix and using batch matrix multiplication makes the inner-loop computation more efficient.
D Ablation study on Repeated MNIST
The Repeated MNIST ablation shows that redundant repetitions increasingly degrade BALD, while BatchBALD maintains its performance and reaches 90% accuracy with 100 data points across repetition settings.
- BatchBALD reaches 90% accuracy with 100 data points for every tested repetition count.
- BALD reaches 90% accuracy at 120 data points with zero repetitions, but requires 160, 280, and 300 data points with 1, 2, and 4 repetitions.
- As repetitions increase, BALD performs worse, whereas BatchBALD outperforms BALD with zero repetitions.
E Additional results for Repeated MNIST
Additional results show that BatchBALD outperforms alternative acquisition methods and selects more diverse, uniformly represented classes on EMNIST than BALD and random acquisition.
- BatchBALD outperforms Var Ratios and Mean STD.
- The Repeated MNIST comparison uses acquisition size 10 and 10 MC dropout samples across BALD, BatchBALD, Var Ratios, Mean STD, and random acquisition.
- EMNIST contains 47 classes represented in the additional-results examples.
- On EMNIST, BatchBALD consistently outperforms random acquisition and BALD, while BALD does not beat random acquisition.
- BatchBALD steadily acquires a more diverse set of EMNIST data points than BALD across acquisition steps.
- BatchBALD acquires EMNIST classes more uniformly, while several classes are underrepresented in BALD and random acquisition.