Source-linked AI summary
Cold-start Active Learning through Self-supervised Language Modeling
Michelle Yuan, Hsuan-Tien Lin, Jordan Boyd-Graber
TL;DR
The paper addresses the difficulty of selecting informative examples for text classification when labeled data are scarce and model confidence is unreliable. It introduces ALPS, which uses BERT’s masked language-modeling loss to estimate uncertainty and combines this signal with diversity-oriented sampling. ALPS outperforms active-learning baselines in accuracy and algorithmic efficiency across four datasets spanning three domains.
Problem
Active learning needs to reduce costly annotation, but cold-start selection is difficult because deep-model confidence is unreliable and training on small datasets is unstable.
Method
ALPS uses BERT’s masked language-modeling loss as a self-supervised proxy for classification uncertainty when selecting the first data batch.
Results
ALPS outperforms active-learning baselines in accuracy and algorithmic efficiency across four text-classification datasets spanning three domains.
Takeaways & Limitations
Pre-trained language-model surprisal can guide sampling toward examples that are both surprising and substantial in cold-start active learning.
Takeaways & Limitations
BADGE-style uncertainty-diversity methods face warm-start and computational-efficiency drawbacks, while small-data fine-tuning makes confidence and inferred labels unreliable.
Abstract
from arXiv · showhide
Active learning strives to reduce annotation costs by choosing the most critical examples to label. Typically, the active learning strategy is contingent on the classification model. For instance, uncertainty sampling depends on poorly calibrated model confidence scores. In the cold-start setting, active learning is impractical because of model instability and data scarcity. Fortunately, modern NLP provides an additional source of information: pre-trained language models. The pre-training loss can find examples that surprise the model and should be labeled for efficient fine-tuning. Therefore, we treat the language modeling loss as a proxy for classification uncertainty. With BERT, we develop a simple strategy based on the masked language modeling loss that minimizes labeling costs for text classification. Compared to other baselines, our approach reaches higher accuracy within less sampling iterations and computation time.
1 Introduction
Active learning addresses costly NLP annotation but is difficult to deploy from a cold start because deep-model confidence and training are unreliable. The paper proposes using pre-trained language-model surprisal as a self-supervised proxy for classification uncertainty and introduces ALPS for efficient sampling.
- Annotation cost and time make labeled data a fundamental machine-learning bottleneck, especially in NLP.
- Deep-model uncertainty sampling is unreliable because neural networks are poorly calibrated, and querying one document per iteration increases training expense.
- Cold-start active learning is needed because conventional strategies depend on task-informed models, classification loss, or confidence scores.
- Pre-trained models provide encoded language knowledge, so annotation can target information missing from pretraining.
- ALPS selects its first batch with BERT’s masked language-modeling loss, using surprising examples as a self-supervised signal for cold-start acquisition.
- Across four text-classification datasets and three domains, ALPS outperforms active-learning baselines in accuracy and algorithmic efficiency.
2 Preliminaries
The paper formalizes sentence-classification active learning as iterative label acquisition from an unlabeled pool, followed by classifier fine-tuning. Its BERT-based setup uses pre-trained representations and a classification head to produce task predictions.
- BERT is pre-trained with masked language modeling and next sentence prediction, and its [CLS] final hidden state serves as the BERT embedding.
- The fine-tuned classifier applies a softmax to a linear transformation of the BERT representation, producing confidence scores for C labels.
- Active learning samples a labeled subset D from an unlabeled sentence pool U so fine-tuning improves test accuracy.
- On each iteration, strategy A acquires k sentences, queries their labels, and after T rounds obtains labels for Tk sentences.
- Cold-start acquisition uses the pre-trained model with parameters θ0 rather than a previous iteration’s task-trained model.
3 The Uncertainty–Diversity Dichotomy
Active learning combines uncertainty, which targets difficult examples, with diversity, which covers heterogeneous feature space. Prior methods integrate these goals but face warm-start, calibration, and computational limitations, motivating a self-supervised alternative.
- Uncertainty and diversity: Uncertainty sampling searches for difficult-to-label examples, whereas diversity sampling exploits heterogeneity in feature space.
- Uncertainty and diversity: Uncertainty sampling requires model warm-starting, while diversity sampling can operate from a cold start.
- BADGE: BADGE combines confidence-based uncertainty with clustering to select diverse and uncertain batches, but retains warm-start and efficiency drawbacks.
- BADGE: BADGE gradient embeddings encode hidden representations scaled by confidence differences and are clustered with k-MEANS++.
- Limitations: Small-data fine-tuning makes confidence scores and model-inferred labels unreliable for warm-start acquisition methods.
- Limitations: High-dimensional distance comparisons can become expensive and less meaningful because of dimensionality effects, including BADGE’s Cd-dimensional embeddings.
4 A Self-supervised Active Learner
ALPS applies self-supervised masked language modeling to cold-start active learning, using surprisal embeddings to estimate uncertainty without classification supervision and clustering them to preserve diversity.
- Motivation: ALPS addresses cold-start active learning by replacing unreliable classification uncertainty estimates with self-supervised language modeling information.Cold-start settings lack sufficient labeled data, while model confidence and inferred labels can be unreliable for poorly trained models.
- Surprisal Embeddings: The method computes surprisal embeddings from BERT’s MLM head using unmasked sentences while evaluating a random 15% of tokens.The evaluated token losses estimate token-context surprisal; unsampled token positions receive zero entries.
- Surprisal Embeddings: Each nonzero surprisal-embedding entry estimates the corresponding token’s surprisal within its sentence context.Surprisal is defined as the negative log likelihood of a token given its context.
- Surprisal Embeddings: Higher-surprisal sentences can be harder for a fine-tuned classifier to label because they are more confusing for the pre-trained model.The paper illustrates this with a sentence containing rare or unusual token combinations that receive higher surprisal.
- Batch Selection: Clustering is included because selecting only the most surprising sentences could reduce generalization and produce redundant atypical examples.The paper identifies diversity as a safeguard against batches dominated by similar rare-token sentences.
- Batch Selection: ALPS clusters surprisal embeddings with k-MEANS and selects the sentence nearest each cluster center for oracle labeling.The resulting batch combines surprisal-based uncertainty with diversity, and experiments report that k-MEANS works better than k-MEANS++ for these embeddings.
5 Active Sentence Classification
The experiments evaluate ALPS against warm- and cold-start baselines across three sentence-classification domains, measuring accuracy over ten iterations and runtime per sampling iteration. ALPS generally achieves the highest early-iteration accuracy, while runtime and convergence differ across strategies and datasets.
- 5 Active Sentence Classification: The simulation queries 100 sentences per iteration for ten iterations, repeatedly fine-tuning from the pre-trained encoder and evaluating test micro-F1.The experiments cover sentiment reviews, news articles, and medical abstracts, using BERT-Base or SCIBERT depending on the dataset.
- 5.1 Baselines: ALPS selects batches using masked language modeling loss, while the baselines include entropy, BADGE, FT-BERT-KM, Random, and BERT-KM.BERT-KM clusters pre-trained BERT embeddings; FT-BERT-KM instead uses embeddings from a previously fine-tuned model.
- 5.3 Results: ALPS-sampled data produces higher test accuracy than the baselines, especially during earlier iterations for AG NEWS, IMDB, and SST-2.Figure 2 compares simulated active learning with the accuracy of a model fine-tuned on the entire dataset.
- 5.3 Results: On PUBMED, ALPS and BADGE achieve about the same accuracy, while naïve strategies perform poorly on the specialized, class-imbalanced dataset.The paper attributes the similar performance to both methods combining uncertainty and diversity, which alleviates class-imbalance problems.
- 5.3 Results: BADGE, FT-BERT-KM, and BERT-KM take much longer per sampling iteration than the alternatives on large datasets.The runtime comparison is reported in Table 2; the paper also gives ALPS complexity as O(tknl).
- 5.3 Results: As sampling iterations increase, test accuracy across methods converges, and ALPS and BADGE approach the full-training-data model across all tasks.The authors frame the first ten iterations as the cold-start period; afterward, uncertainty-based methods can query more confusing examples.
6 Analyzing ALPS
ALPS samples increasingly diverse batches while maintaining uncertainty, and surprisal embeddings support effective clustering and flexible sampling. It also matches BADGE’s accuracy more quickly and can query all requested sentences in one iteration.
- Sampling Efficiency: ALPS reaches BADGE’s accuracy more quickly: average runtime per iteration is 24 minutes for ALPS versus 70 minutes for BADGE on PUBMED.The stated complexities are O(tknl) for ALPS and O(Cknd) for BADGE.
- Diversity and Uncertainty: Diversity is measured by token overlap with unsampled data, while uncertainty is measured by average predictive entropy from a stable classifier.Higher token-overlap similarity indicates better diversity and representation.
- Diversity and Uncertainty: ALPS increases sample diversity with minimal uncertainty loss on AG NEWS and achieves the highest diversity among algorithms in the final PUBMED iteration.Other methods show larger uncertainty drops as iterations increase.
- Surprisal Clusters: Surprisal embeddings have a Silhouette Coefficient of 0.38, compared with 0.04 for BERT embeddings.The corresponding t-SNE plots suggest that surprisal embeddings are easier to cluster, although neither embedding set clearly separates classification labels.
- Surprisal Clusters: The MLM head maps BERT representations into surprisal space, where simple k-MEANS can efficiently select representative sentences.This avoids relying on more complex clustering algorithms that would increase algorithmic complexity and runtime.
- Single-iteration Sampling: ALPS produces little accuracy difference between iterative sampling and querying all k sentences in one iteration.This flexibility supports settings with costly retraining or unpredictable annotation availability.
7 Related Work
Prior active-learning research includes uncertainty, diversity, transfer-learning, Bayesian, reinforcement-learning, and batch methods. A central open question is how to combine uncertainty and diversity effectively.
- Prior Applications: Active learning has been applied to named entity recognition, word sense disambiguation, sentiment analysis, semantic parsing, and machine translation.The cited work spans both traditional and neural models.
- Transfer Learning: Transfer-learning approaches select informative target-domain texts, match conditional distributions, or combine word-level and document-level annotations.These methods aim to support knowledge transfer across domains or annotation levels.
- Other Approaches: Other deep active-learning work explores Bayesian and reinforcement-learning approaches, with integration into ALPS identified as a future direction.The passage presents this integration as an interesting research direction rather than an evaluated result.
8 Conclusion
The paper addresses annotation-efficient fine-tuning by using pretrained transformer knowledge in a cold-start active-learning strategy. ALPS uses self-supervision to select diverse and uncertain examples, while remaining flexible and computationally efficient.
- 8 Conclusion: ALPS uses the pretrained loss to sample diverse and uncertain examples without relying on classification supervision.The method projects data into an embedding space and selects representative points using self-supervision.
- 8 Conclusion: The approach is designed for limited-annotation settings where transformer accuracy and stability depend on fine-tuning data.The conclusion frames labels as information that should complement rather than duplicate unsupervised data.
- 8 Conclusion: Future work may seek representations that encode the information most important for active learning.This is presented as a direction for improving representation quality.
A Appendices
The appendix compares k-MEANS and k-MEANS++ for selecting centroids in surprisal embeddings. k-MEANS achieves higher validation accuracy.
- A Appendices: Using k-MEANS to select surprisal-embedding centroids reaches higher validation accuracy than k-MEANS++.
A.1 Token Masking
ALPS evaluates inputs in the same unmasked format used for fine-tuning, because masking does not help when token embeddings are not being trained to learn context.
- A.1 Token Masking: Unmasked inputs improve validation accuracy compared with passing the original input with masks.The comparison is reported in preliminary validation experiments.
- A.1 Token Masking: ALPS uses the same input format for active learning and fine-tuning to avoid a mismatch between the two stages.
A.2 Token Sampling for Evaluation
The evaluation examines token-sampling choices and diversity in ALPS. Results indicate that sampling percentage matters mainly early, while k-MEANS and surprisal embeddings support more diverse batches than competing choices or random sampling.
- A.2 Token Sampling for Evaluation: 15% random token sampling is the BERT pretraining default, while experiments compare 10%, 15%, 20%, and 100%.BERT’s MLM loss focuses only on the sampled masked tokens.
- A.2 Token Sampling for Evaluation: Mean accuracy is roughly similar across token-sampling percentages, but non-15% settings show slightly higher variance across runs.
- A.2 Token Sampling for Evaluation: After the second AL iteration, accuracy means and variances converge across token-sampling percentages, making the percentage more influential in early AL stages.
- A.2 Token Sampling for Evaluation: Mask-strategy accuracy differences are minimal for BERT fine-tuning, a pattern the authors believe also applies to ALPS.
- A.2 Token Sampling for Evaluation: k-MEANS on surprisal embeddings produces larger accuracy gains than k-MEANS++, especially for PubMed, and selects centers farther apart.The k-MEANS result is contrasted with k-MEANS++ and random sampling in the reported experiments.
- A.2 Token Sampling for Evaluation: ALPS samples more diverse content than random sampling and avoids repeatedly selecting sentences with similar surprisal patterns.The authors suggest this may diversify syntactic structure in sampled batches.