Source-linked AI summary
Practical Obstacles to Deploying Active Learning
David Lowell, Zachary C. Lipton, Byron C. Wallace
TL;DR
The paper examines whether AL’s gains generalize reliably across models and tasks, and whether actively acquired datasets remain useful for successor models. Across the experiments, gains are modest and inconsistent, while transfer to successor models is often poor.
Problem
Method
The study evaluates standard AL across text classification and sequence tagging, and tests training successor models on datasets acquired by different models.
Source: §7 · p. 9Results
In reported classification datapoints, standard AL beats i.i.d. sampling in 60.9% of cases, while actively acquired datasets often underperform when transferred to successor models.
Source: §6 · p. 8Takeaways & Limitations
AL’s gains are marginal and inconsistent, and an acquisition strategy’s success for a particular model and task cannot be predicted clearly ahead of time.
Source: §7 · p. 9Actively acquired datasets may be disadvantageous for training subsequent models.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget. In AL one iteratively selects training examples for annotation, often those for which the current model is most uncertain (by some measure). The hope is that active sampling leads to better performance than would be achieved under independent and identically distributed (i.i.d.) random samples. While AL has shown promise in retrospective evaluations, these studies often ignore practical obstacles to its use. In this paper we show that while AL may provide benefits when used with specific models and for particular domains, the benefits of current approaches do not generalize reliably across models and tasks. This is problematic because in practice one does not have the opportunity to explore and compare alternative AL strategies. Moreover, AL couples the training dataset with the model used to guide its acquisition. We find that subsequently training a successor model with an actively-acquired dataset does not consistently outperform training on i.i.d. sampled data. Our findings raise the question of whether the downsides inherent to AL are worth the modest and inconsistent performance gains it tends to afford.
1 Introduction
Active learning (AL) aims to reduce annotation needs by selecting examples for labeling, but its non-i.i.d. datasets and uncertain performance complicate real-world use. The paper reports that gains are often marginal and inconsistent, and acquired datasets do not consistently transfer to successor models.
1 Introduction
- Because labels are expensive, deep learning’s strong supervised results require large annotated datasets, motivating methods that reduce supervision.Source: §1 · p. 1
- Pool-based AL iteratively scores unlabeled instances and selects high-scoring examples for annotation instead of relying on random sampling.Source: §1 · p. 1
- Practitioners cannot readily validate an acquisition strategy without additional i.i.d. data, whose collection would defeat AL’s purpose of reducing supervision.Source: §1 · p. 2
- Common AL algorithms behave inconsistently across models and datasets, often performing no better than i.i.d. sampling.A strategy that outperforms i.i.d. sampling on one dataset may differ from the strategy that wins on another.Source: §1 · p. 2, Figure 1 · p. 2
- AL typically affords only marginal gains at the cost of non-i.i.d. training samples that do not consistently transfer to later models.Source: §1 · p. 2
2 The (Potential) Trouble with AL
AL’s gains vary across datasets, and the training data it acquires can be coupled to the model that selected it. This raises practical questions about whether those datasets help successor models when they outlive the acquisition model.
2 The (Potential) Trouble with AL
- Across four datasets, the relative gains from BiLSTM maximum-entropy sampling over random sampling are decidedly mixed.Positive, zero, and negative values indicate whether AL, neither strategy, or random sampling performs better.Source: §2 · p. 2
- Acquisition functions generally depend on the model being trained, coupling the acquired dataset to its acquisition model even though datasets often outlast models.Source: §2 · p. 2
- The paper asks how a successor model trained on another model’s acquired data compares with training on native or i.i.d. data.Source: §2 · p. 3
- On Subjectivity, a BiLSTM trained with data acquired by either a CNN or an SVM performs worse than with i.i.d. sampling.Source: §2 · p. 3
- Prior work studied transferability of actively acquired datasets, but did not examine modern deep-learning models or the paper’s broader practical focus.Source: §2 · p. 3
3 Experimental Questions and Setup
The experiments test AL’s reliability and dataset transfer across text classification and named-entity recognition. They compare acquisition and successor models, then evaluate performance against native and i.i.d. training data.
3 Experimental Questions and Setup
- The experiments ask whether AL reliably beats i.i.d. sampling and whether data acquired by one model can train a different successor, using classification and NER tasks.Source: §3 · p. 3
- The study includes linear models and deep networks, examining both same-model AL and every acquisition–successor model pair.Source: §3 · p. 3
- Each acquisition model is warm-started with i.i.d. labeled data, then iteratively selects instances from an unlabeled pool for annotation.Source: §3 · p. 3
- Successor performance is evaluated on a held-out test set and compared with native and i.i.d. training sets of comparable size across learning curves.Source: §3 · pp. 3–4
4 Tasks
This section describes the classification and NER tasks, their models and data, and the acquisition functions used to select examples. It also identifies the experiments’ main task-specific reporting formats.
4 Tasks
- The experiments cover text classification and NER, with descriptions of their models, datasets, acquisition functions, and implementation details.The section covers text classification in 4.1 and NER in 4.2.Source: §4 · p. 4
4.1 Text Classification
- Classification experiments pair SVMs, CNNs, and BiLSTMs with four corpora; each experiment uses a shared initial labeled set sampled at 2.5% of the remaining pool.The corpora are Movie Reviews, Subjectivity, TREC, and Customer Reviews.Source: §4.1 · p. 4
4.2 Named Entity Recognition
- NER experiments transfer between CRFs and BiLSTM-CNNs on CoNLL-2003 and OntoNotes, merging training and validation data into the pool and initializing each labeled set at 2.5%.The CRF uses word-level and character-based features, while the BiLSTM-CNN uses pretrained GloVe vectors and learned word and character features.Source: §4.2 · p. 4
4.3 Acquisition Functions
- The acquisition methods include uncertainty sampling, QBC, and BALD; classification uncertainty uses entropy, while NER MNLP normalizes predicted tag-sequence log probabilities by sequence length.For SVM classification, uncertainty sampling selects documents closest to the decision boundary; MNLP uses greedy decoding to estimate the most likely tag sequence.Source: §4.3 · p. 5
Text classification
- Table 1 reports classification accuracy for acquisition–successor model combinations at 10% and 20% of the document pool, with colors indicating performance relative to i.i.d. baselines.Blue means better than the i.i.d. baseline, red worse, and black the same.Source: Table 1 · p. 6
Named Entity Recognition
- For NER, Table 2 reports F1 at 10% and 20% of the training pool, while Figure 2 plots sample learning curves by successor model and acquisition model against a blue i.i.d. baseline.Figure 2 shows examples for Movie Reviews and OntoNotes using maximum-entropy acquisition.Source: Figure 2 · p. 5, Table 2 · p. 6
4.3 Acquisition Functions
- QBC uses bagged committees of 10 models, with Kullback-Leibler disagreement for classification and average per-word vote entropy for NER; OntoNotes is excluded because committee training is impractical.The committee’s bootstrap samples each contain as many documents as the current training set.Source: Named Entity Recognition · pp. 6–7
- BALD estimates uncertainty from disagreement across multiple dropout passes, and its dropout dependence rules it out for non-neural SVM and CRF models.The method applies dropout at test time and selects instances with disagreement across passes.Source: Named Entity Recognition · p. 7
5 Results
The results compare acquisition and successor models across tasks, using tables and learning curves to report performance. Additional results and the points used to compare learning curves are specified.
5 Results
- For each task, the study compares every acquisition–successor model pair using performance tables and learning curves over training-set size.Tables 1 and 2 cover classification and NER, respectively; Figure 2 shows the learning curves.Source: §5 · p. 7
- Appendix results add learning curves and acquisition-function tables, plus plots comparing standard AL with corresponding i.i.d. data.Classification is evaluated with accuracy and NER with F1.Source: §5 · p. 7
- Learning-curve comparisons report performance at training-set sizes of 10% and 20% of the training pool.Source: §5 · p. 7
6 Discussion
Discussion finds that active learning’s classification gains are modest and its transferred datasets often fail to beat i.i.d. data. NER results are more favorable, while learned representations remain similar across acquisition strategies.
6 Discussion
- 60.9% of reported classification datapoints show standard AL outperforming i.i.d. sampling, although some acquisition function beats i.i.d. in 75% of model, dataset, and training-size combinations.Source: §6 · p. 8
- 37.5% of classification dataset-transfer datapoints outperform the i.i.d. baseline, and foreign actively acquired datasets tend to underperform i.i.d. data.Transfer here means acquisition and successor models differ.Source: §6 · p. 8
- NER results are more favorable: AL consistently improves over the i.i.d. baseline in both standard and transferred data points.Source: §6 · p. 8
- Classification outcomes depend on model and dataset: uncertainty with SVM or CNN and BALD with CNN show regular gains, but no transfer combination does so consistently.AL performs more reliably on Subjectivity than other datasets and consistently poorly on TREC.Source: §6 · p. 8
- Embedding initialization can materially affect AL results: using word2vec rather than GloVe explained much of an initially observed discrepancy.Source: §6 · p. 8
- Despite predictive-performance differences, transferred datasets yield learned representations similar to those from native active learning.Table 4 reports average Spearman rank correlations of cosine distances between test-set representations.Source: Table 4 · p. 8, §6 · p. 9
7 Conclusions
Across models and datasets, AL's gains are marginal and inconsistent, and acquired data may not serve successor models well. Predicting which AL setup will work is difficult without additional i.i.d. data that would undermine AL's purpose.
7 Conclusions
- The evaluation spans standard AL methods across models, domains, acquisition functions, and text classification and sequence-tagging tasks, including transfer to distinct successor models.The paper treats successor-model use as an anticipated scenario because annotated data often outlives models.Source: §7 · p. 9
- AL's relative success is difficult to predict ahead of time, and assessing strategies requires substantial i.i.d. data that would undermine AL's purpose.Early-cycle success also does not necessarily predict later-cycle success.Source: §7 · p. 9
- Training successor models on actively sampled data often performs worse than using an equivalently sized i.i.d. sample; results are more favorable for NER than text classification.The NER comparison is reported as consistent with prior work (Tomanek and Morik, 2011).Source: §7 · p. 9
- AL often yields only marginal, inconsistent gains over i.i.d. sampling, while its acquired dataset may disadvantage subsequent models.Together, these findings raise concerns about AL's efficacy in practice.Source: §7 · p. 9
A Experimental Results
The appendix reports full experimental results through tables and learning curves, including metrics for multiple acquisition and successor-model combinations at two pool fractions.
A Experimental Results
- Tables 5 and 6 report metrics for source, successor, and acquisition-function combinations at 10% and 20% of the pool, while Figure 3 shows learning curves.Table 7 reports average Spearman’s rank correlation coefficients.Source: A Experimental Results · p. 11
Text classification
Table 5 presents text-classification accuracy across acquisition and successor models, with results reported for two document-pool fractions and compared with i.i.d. baselines.
Text classification
- Table 5 reports text-classification accuracy for uncertainty sampling, QBC, and BALD at 10% and 20% of the document pool, colored by performance relative to i.i.d. baselines.Blue denotes better performance, red worse, and black the same as the baseline.Source: Table 5 · p. 12
Named Entity Recognition
The NER results include F1 measurements at two training-pool fractions, transfer-related representation correlations, and learning curves for classification and NER.
Named Entity Recognition
- Table 6 reports NER F1 measurements for training sets comprising 10% and 20% of the training pool.Source: Table 6 · p. 13
- Table 7 reports average Spearman rank correlations between test-set representation distances from native active learning and transferred actively acquired datasets.Source: Table 7 · p. 13
- Figure 3 contains the collected learning curves for text classification and NER, with error bars representing one standard deviation.Source: Figure 3 · p. 14