Source-linked AI summary
Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study
Aditya Siddhant, Zachary C. Lipton
TL;DR
Deep active learning for NLP lacks reliable evidence across unseen tasks, datasets, and models, which matters because one run can exhaust the annotation budget. The paper conducts a large-scale comparison with honest warm-start hyper-parameter selection and finds that Bayesian BALD methods consistently perform best across broad settings.
Problem
The practical reliability of active learning for unseen NLP problems remains uncertain because a run exhausts its labeling budget and prior studies cover limited task and model combinations.
Method
The paper evaluates acquisition functions across Sentiment Classification, Named Entity Recognition, and Semantic Role Labeling using multiple datasets and models, with hyper-parameters set on warm-start data.
Results
Across many tasks, classic uncertainty sampling and Bayesian approaches outperform i.i.d. baselines, while DO-BALD and BB-BALD consistently perform best.
Takeaways & Limitations
Bayesian active learning by disagreement is effective across combinations of task, dataset, and model, usually outperforming classic uncertainty sampling.
Takeaways & Limitations
For sequence labeling, whole-sequence agreement may bias acquisition toward longer sentences, although word-based budgeting prevents this from constituting an unfair advantage.
Abstract
from arXiv · showhide
Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each against a validation set before selecting a model, AL affords no such luxury. Over the course of one AL run, an agent annotates its dataset exhausting its labeling budget. Thus, given a new task, an active learner has no opportunity to compare models and acquisition functions. This paper provides a large scale empirical study of deep active learning, addressing multiple tasks and, for each, multiple datasets, multiple models, and a full suite of acquisition functions. We find that across all settings, Bayesian active learning by disagreement, using uncertainty estimates provided either by Dropout or Bayes-by Backprop significantly improves over i.i.d. baselines and usually outperforms classic uncertainty sampling.
1 Introduction
Deep active learning promises to reduce annotation needs, but reliable uncertainty estimation and out-of-the-box performance on unseen NLP problems remain open challenges. This study evaluates Bayesian acquisition methods broadly rather than championing one approach.
- Deep learning’s data dependence makes restricted annotation budgets a formidable obstacle, motivating Active Learning to select examples strategically.
- For practical active learning, methods must work on unseen datasets without exhausting the labeling budget through poor acquisition choices.The paper also highlights target leakage when architectures and hyper-parameters are tuned using concealed labels.
- The study compares acquisition functions across Sentiment Classification, Named Entity Recognition, and Semantic Role Labeling, using multiple datasets and models.Hyper-parameters are set on warm-start data for a more honest assessment.
- BALD with either Dropout or Bayes-by-Backprop uncertainty proved effective across all combinations of task, dataset, and model.The two variants performed similarly well across most settings.
- Prior deep active-learning NLP studies generally examined only one task and one model, despite using multiple datasets.
2 Bayesian Deep Learning
The paper compares Bayesian uncertainty estimates for active learning, including prediction-time Monte Carlo Dropout and Bayes-by-Backprop’s learned weight distributions.
- Monte Carlo Dropout: Monte Carlo Dropout estimates uncertainty by comparing outputs from T stochastic prediction passes through a trained neural network.
- Bayes by Backprop: Bayes-by-Backprop represents each network weight with a probability distribution instead of a point estimate.
- Bayes by Backprop: Bayes-by-Backprop approximates the posterior p(w|D) with a variational distribution q(w|θ), typically a Gaussian with diagonal covariance.
- Bayes by Backprop: The variational parameters include weight means µi and softplus-parameterized standard deviations σi = log(1+exp(ρi)).
- Bayes by Backprop: The variational objective minimizes the KL divergence between the approximate distribution and the posterior, optimized with stochastic gradient descent and re-parameterization.
3 Experimental Setup
The experiments compare classical and Bayesian uncertainty-based acquisition across three NLP tasks, multiple datasets and architectures, and progressively larger annotation budgets. Bayesian BALD methods generally outperform random and classic uncertainty baselines, with substantial gains on NER.
- Acquisition functions: The study compares least confidence for classification and maximum length-normalized log probability for sequence labeling.MNLP normalizes sequence log probability by length to remove preference for longer sequences.
- Acquisition functions: BALD selects examples where stochastic forward passes disagree most often with the plurality prediction; the variants are DO-BALD and BB-BALD.
- Acquisition functions: For sequence tasks, agreement is measured over the entire sequence tag, which may bias selection toward longer sentences.The annotation budget is measured in words rather than sentences.
- Active-learning process: Each active-learning run begins with 2% random warm-start data, adds 2% per round, retrains from scratch, and stops at 50% annotation.
- Datasets and models: The evaluation covers SC, NER, and SRL across two datasets per task, with multiple model architectures and annotation budgets.Figures 1, 2, and 3 report performance for SC, NER, and SRL, respectively.
- Results: Across the experiments, active-learning methods outperform random i.i.d. labeling, while DO-BALD and BB-BALD consistently outperform classic uncertainty sampling.The improvement over classic uncertainty sampling is marginal in a few cases.
- Results: 98-99% of full-dataset NER performance is achieved with 20% of samples, whereas the i.i.d. baseline requires 50% for comparable F score.This result holds for both CNN-BiLSTM-CRF and CNN-CNN-LSTM models.
4 Conclusion
The study finds a consistent practical advantage for Bayesian active learning across many NLP settings, without label peeking for hyperparameter tuning.
- 4 Conclusion: Over 40 experiments were each repeated three times, with roughly 25 retraining rounds and about 3000 training runs overall.The scale was designed to assess whether active learning has a consistent practical story.
- 4 Conclusion: Classic uncertainty sampling and Bayesian approaches both outperform i.i.d. baselines across many tasks.
- 4 Conclusion: DO-BALD and BB-BALD consistently perform best across the evaluated settings.