Source-linked AI summary
Deep Bayesian Active Learning with Image Data
Yarin Gal, Riashat Islam, Zoubin Ghahramani
TL;DR
Active learning needs models that learn from small labelled datasets and represent uncertainty, but deep learning typically depends on large datasets and rarely models uncertainty. The paper combines Bayesian deep learning with active learning through Bayesian convolutional neural networks for high-dimensional images, reporting improved data efficiency on MNIST and lesion diagnosis. Its experiments reduce required expert labels, although repeated retraining to convergence can require prolonged runtimes.
Problem
Active learning lacks scalable approaches for high-dimensional images because suitable models must learn from small datasets and represent prediction uncertainty.
Method
The paper combines Bayesian deep learning and active learning using Bayesian convolutional neural networks, uncertainty-based acquisition functions, and dropout-based approximate inference.
Results
5% test error on MNIST requires 295 labelled images with the proposed system versus 835 with random sampling; BALD also achieves better AUC faster than uniform selection on lesion images.
Takeaways & Limitations
The techniques reduce the number of expert labels required for image-based active learning, including medical diagnosis.
Takeaways & Limitations
Resetting and retraining models to convergence isolates acquisition-function effects but causes prolonged training times, such as 20 hours per melanoma experiment.
Abstract
from arXiv · showhide
Even though active learning forms an important pillar of machine learning, deep learning tools are not prevalent within it. Deep learning poses several difficulties when used in an active learning setting. First, active learning (AL) methods generally rely on being able to learn and update models from small amounts of data. Recent advances in deep learning, on the other hand, are notorious for their dependence on large amounts of data. Second, many AL acquisition functions rely on model uncertainty, yet deep learning methods rarely represent such model uncertainty. In this paper we combine recent advances in Bayesian deep learning into the active learning framework in a practical way. We develop an active learning framework for high dimensional data, a task which has been extremely challenging so far, with very sparse existing literature. Taking advantage of specialised models such as Bayesian convolutional neural networks, we demonstrate our active learning techniques with image data, obtaining a significant improvement on existing active learning approaches. We demonstrate this on both the MNIST dataset, as well as for skin cancer diagnosis from lesion images (ISIC2016 task).
1. Introduction
Active learning aims to reduce the costly burden of obtaining labelled data by selecting informative examples for an external oracle. The paper addresses the difficulty of extending this framework to high-dimensional images with models that learn from little data and represent uncertainty.
- Active learning trains on a small initial set, selects unlabelled pool points through an acquisition function, obtains oracle labels, and repeats.The training set grows after each acquisition cycle.
- High-dimensional image data remains difficult for active learning because models must learn from small datasets while representing uncertainty over unseen data.These requirements restrict the models usable within the framework and have led most approaches to focus on low-dimensional problems.
- Deep learning is not prevalent in active learning because it typically depends on large amounts of data and rarely represents model uncertainty.
- The paper combines Bayesian deep learning with active learning to develop a practical framework for high-dimensional image data using Bayesian convolutional neural networks.The authors demonstrate the approach on MNIST and skin-lesion images.
- 5% test error on MNIST requires 295 labelled images with the proposed system, compared with 835 images using random sampling.The proposed system also achieves 1.64% test error with 1000 labelled images.
2. Related Research
Prior image-based active learning work primarily used kernel methods to model image similarity or uncertainty, while semi-supervised learning used fixed labelled and unlabelled sets rather than iterative querying.
- Earlier image active learning approaches used SVMs with linear, polynomial, or RBF kernels and probabilistic outputs for margin-based uncertainty.
- Gaussian-process and Gaussian-random-field methods also used RBF kernels to obtain uncertainty or select points from raw images.The Gaussian-random-field approach additionally used unlabelled data.
- Semi-supervised image learning gives a model fixed labelled and unlabelled datasets, using the unlabelled inputs to learn about input distributions.This learning paradigm differs from active learning, which selects new points for labelling.
3. Bayesian Convolutional Neural Networks
The paper uses Bayesian convolutional neural networks to combine image-specific representations with uncertainty estimates suitable for active learning. Dropout supplies a practical approximate Bayesian inference procedure through stochastic test-time forward passes.
- CNNs are used instead of kernel methods because the paper targets high-dimensional image data and requires prediction uncertainty.
- The model defines classification predictions with a softmax likelihood p(y = c|x, ω) = softmax(f_ω(x)) and can use a Gaussian likelihood for regression.
- Dropout is applied before weight layers during training and at test time to sample an approximate posterior through stochastic forward passes, called MC dropout.
- Dropout can be interpreted as variational Bayesian inference that approximates the model posterior by minimising KL divergence within a tractable distribution family.
- Bayesian CNNs can work with small amounts of data and provide uncertainty information for existing acquisition functions.
4. Acquisition Functions and their Approximations
An acquisition function scores unlabelled pool points using the model and selects the point with the highest score. The paper discusses predictive-variance-based selection for regression and analogous uncertainty-driven choices for images.
- An acquisition function a(x, M) scores each input x in the unlabelled pool using model M to decide which point to query next.
- The next queried point is x∗ = argmax_x∈Dpool a(x, M), selecting the pool input with the largest acquisition value.
- For regression, predictive variance or a derived quantity can guide acquisition, but selection should avoid choosing points that are merely noisy.
1. Choose pool points that maximise the predictive en-
The paper evaluates acquisition functions that select pool points using predictive or model uncertainty, approximating Bayesian criteria with an approximate posterior. BALD, Variation Ratios, and Max Entropy are contrasted with Mean STD and random acquisition.
- BALD: BALD selects points with high mutual information between predictions and model parameters.These points are uncertain on average while allowing model parameters to produce disagreeing, individually confident predictions.
- BALD: BALD points correspond to high variance in the logits, so stochastic forward passes tend to assign the highest probability to different classes.
- Other acquisition functions: Variation Ratios and Max Entropy measure predictive lack of confidence, while Mean STD averages standard deviation across possible classes.Mean STD is described as an ad-hoc technique relative to the other acquisition functions.
- Approximation: The acquisition functions are approximated using an approximate posterior, yielding a computationally tractable BALD estimator.The other acquisition functions can be approximated similarly.
- Experimental comparison: The experiments compare BALD, Variation Ratios, Max Entropy, Mean STD, and Random against one another and other image-data active-learning techniques.The evaluation includes a random pool-acquisition baseline and a real-world cancer-diagnosis case study.
5. Active Learning with Bayesian Convolutional Neural Networks
The paper evaluates Bayesian CNN-based active learning for image data, comparing uncertainty-driven acquisition functions with deterministic, kernel-based, semi-supervised, and random baselines. Experiments on MNIST and melanoma images show improved data efficiency and early performance, while also exposing dataset-specific limitations.
- MNIST acquisition experiments: Bayesian CNN uncertainty acquisition functions are evaluated on MNIST against random acquisition and a deterministic CNN using repeated pool-based image selection.The experiments acquire images iteratively and assess test performance after each acquisition.
- MNIST acquisition experiments: Variation Ratios, BALD, and Max Entropy outperform Random and Mean STD, with Variation Ratios obtaining slightly better accuracy faster than the other leading functions.Mean STD performs similarly to Random in the reported MNIST experiment.
- MNIST acquisition experiments: BALD, Variation Ratios, and Max Entropy reach 5% and 10% test error with fewer acquisitions than Mean STD and Random, demonstrating improved data efficiency.The paper states that Variation Ratios can require less than half as many labelled images as random acquisition in the example comparison.
- Importance of model uncertainty: Bayesian models attain higher accuracy earlier and converge to higher overall accuracy than deterministic models for BALD, Variation Ratios, and Max Entropy.The comparison attributes this difference to uncertainty propagated through the Bayesian models and its effect on confidence estimates.
- Comparison with existing image-data methods: On two-digit MNIST classification, even Random acquisition with a CNN outperforms MBR based on an RBF kernel, while replacing that kernel with a CNN does not improve MBR.The comparison evaluates BALD, Variation Ratios, Max Entropy, and Random alongside MBR.
- Comparison with semi-supervised learning: With 1000 labelled images, the Var Ratio model reaches 1.64% error without additional unlabelled data, close to the Γ-model’s 1.53% error using semi-supervised learning.The paper notes that the ladder network reaches 0.84% error with 1000 labelled and 59,000 unlabelled images, while its own model is comparatively small.
- Cancer diagnosis from lesion image data: In melanoma diagnosis, BALD achieves better AUC faster than uniform acquisition and selects more positive examples at each acquisition step.The experiments use fixed test splits, repeat each experiment three times, and average performance with standard errors.
- Cancer diagnosis from lesion image data: Variation Ratios fails on melanoma data because malignant and benign images receive nearly indistinguishable probabilities, giving all pool points identical acquisition values.This contrasts with its strong MNIST performance and illustrates that acquisition behavior depends on the task and model outputs.
6. Future Research
The paper presents Bayesian deep-learning active learning for image data and demonstrates it in medical diagnosis. It reduces required expert labels, while reset-and-retrain evaluation prolongs training and avoiding resets may risk local optima.
- The paper presents a new approach for active learning with image data at the intersection of Bayesian modelling and deep learning.
- Reducing required expert labels reduces system costs, but resetting models after each acquisition requires prolonged training.Each melanoma experiment took 20 hours under the reset-and-retrain evaluation.
- Not resetting the system could reduce running time, with the potential price of falling into local optima.