PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Importance of Effectively Adapting Pretrained Language Models for Active Learning

Katerina Margatina, Loïc Barrault, Nikolaos Aletras

arXiv:2104.08320v2cs.CL

TL;DR

NLP Active Learning has adopted pretrained language models, but standard per-iteration fine-tuning is unstable across its low- and high-resource regimes. The paper adapts the LM using all unlabeled task data and introduces robust fine-tuning, finding substantially better results than standard fine-tuning across five datasets.

  • Problem

    Pretrained LMs used in NLP active learning are not adapted effectively to downstream tasks, while standard fine-tuning is unstable, especially with limited data.

    Source: §1 · p. 1

  • Method

    The method continues LM pretraining on all unlabeled task data and uses a fine-tuning strategy designed for both low- and high-resource AL settings.

    Source: §1 · p. 1, §5 · p. 5

  • Results

    The approach yields substantially better results than standard fine-tuning across five standard NLP datasets.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    LM training strategy can be more important than the acquisition strategy for active learning effectiveness.

    Source: §1 · p. 2, §5 · p. 5

    The paper leaves evaluation of larger LMs such as GPT-3 and FLAN for future work.

    Source: §5 · p. 5

Abstract

from arXiv · show

Recent Active Learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.

1 Introduction

Active Learning uses iterative data selection to train models efficiently when unlabeled data is abundant but annotation is limited. In NLP, pretrained LMs are promising but standard per-iteration fine-tuning is unstable across AL’s low- and high-resource settings, motivating task adaptation and robust training.

Active Learning background

  • AL alternates between training on labeled data and selecting unlabeled points for annotation until a budget or performance criterion is reached.Acquisition functions rank unlabeled examples using informativeness signals such as uncertainty, diversity, or both.Source: §1 · p. 1

Motivation

  • Pretrained language models generally outperform task-specific models trained from scratch, but their standard AL fine-tuning remains unstable, especially on small datasets.Recent NLP AL work uses pretrained LMs, yet commonly trains them for a fixed number of epochs at each iteration.Source: §1 · p. 1

Contribution

  • The proposed approach continues LM pretraining on the entire unlabeled pool, then applies a fine-tuning method designed for both low- and high-resource AL settings.This uses more available task data than relying only on labels acquired at each iteration.Source: §1 · p. 1

Results

  • Across five standard NLU tasks and multiple acquisition functions, the approach outperforms all baselines.The analysis further indicates that LM adaptation strategy can matter more than the acquisition strategy.Source: §1 · p. 2

2 Adapting & Fine-tuning Pretrained Models for Active Learning

The AL setup begins with an unlabeled pool, a pretrained LM, an annotation budget, and an acquisition function. The method first performs task-adaptive pretraining on the pool, then initializes each AL model from the adapted weights and fine-tunes it with labeled data.

AL setup

  • The AL protocol selects batches of k unlabeled points from Dpool with an acquisition function until the annotation budget is exhausted, then evaluates on Dtest.The setup includes a pretrained LM, model weights, acquisition size, and a fixed number of AL iterations.Source: §2 · p. 2

Adaptation (TAPT)

  • Task-adaptive pretraining fine-tunes the pretrained LM on all unlabeled downstream-task data before AL iterations begin.The adapted model is then reused as the initialization for subsequent AL classifiers.Source: §2 · p. 2

Fine-tuning (FT+)

  • Each AL iteration adds a classification layer above the adapted BERT representation and fine-tunes the resulting model on the currently labeled data.The classifier uses the [CLS] token representation and jointly updates the adapted model and classification weights.Source: §2 · p. 2

1 Dlab ←∅

The AL algorithm initializes the labeled set as empty and removes each acquired batch from the unlabeled pool as iterations proceed.

1 Dlab ←∅

  • The fine-tuning procedure uses bias correction alongside early stopping, 20 epochs, learning rate 2e-5, and five evaluation steps per epoch.Warmup is limited to min(10% of total steps, 100) to accommodate changing training-set sizes across AL iterations.Source: §1 · p. 2

3 Experiments & Results

Across five diverse language-understanding tasks, TAPT-FT+ improves active-learning data efficiency over standard BERT fine-tuning, especially early in acquisition. The results also suggest training strategy can matter more than acquisition strategy.

Experimental Setup

  • The experiments use five varied language-understanding datasets spanning question classification, sentiment analysis, and topic classification, with binary and multiclass labels.Source: §3 · pp. 2–3

Results

  • TAPT-FT+ reaches full-dataset performance within a 15% annotation budget on all datasets, unlike standard BERT active learning.It reaches the target with almost 10% annotated data on TREC-6 and in the first 2% iteration on DBPEDIA.Source: §3 · p. 3
  • Across all five datasets, adding TAPT and FT+ produces large gains, particularly during the first active-learning iterations.Source: §3 · p. 4
  • After the first 2% acquisition, TAPT+FT+ with ENTROPY exceeds the BERT-SFT ENTROPY baseline by 45, 30, and 12 accuracy points on TREC-6, DBPEDIA, and IMDB.Source: §3 · p. 4

Training vs. Acquisition Strategy

  • Performance curves for acquisition functions are generally close, suggesting training strategy may affect active-learning performance more than acquisition strategy; ENTROPY generally performs best, followed by BADGE.A universally strong acquisition function independent of training strategy remains an open question.Source: §3 · p. 4

4 Analysis & Discussion

The analysis examines why task-adaptive pretraining and resource-aware fine-tuning improve active learning. It finds dataset-dependent TAPT behavior, resource-dependent fine-tuning needs, and complementary gains from both steps.

4.1 Task-Adaptive Pretraining

  • TAPT’s masked-LM loss decreases steadily on DBPEDIA, IMDB, and AGNEWS, correlating with their strong early active-learning gains.Source: §4.1 · p. 4
  • TAPT overfits on TREC-6 and SST-2, attributed to TREC-6’s small dataset and SST-2’s short, less diverse sentences.SST-2 averages 9.4 words per sentence, which may limit the diversity needed for language-model adaptation.Source: §4.1 · p. 4

4.2 Few-shot Fine-tuning

  • Standard fine-tuning is suboptimal with 100 examples, while fewer epochs often work better with 1,000 examples, so no fixed epoch count is uniformly optimal.This motivates a fine-tuning policy that adapts to each active-learning iteration’s resource setting.Source: §4.2 · pp. 4–5

4.3 Ablation Study

  • FT+ improves accuracy and uncertainty calibration on TREC-6, while matching standard fine-tuning on the larger AGNEWS dataset.The strategy remains robust despite substantially increasing warmup steps, covering both low- and high-resource settings.Source: §4.3 · p. 5
  • TAPT combined with FT+ outperforms FT+ without TAPT on both TREC-6 and AGNEWS, especially in early iterations.The result supports using the entire unlabeled pool for additional task-adaptive pretraining.Source: §4.3 · p. 5

5 Conclusion

The paper proposes adapting pretrained LMs with unlabeled task data and robust fine-tuning across resource levels for active learning. Experiments show training strategy can matter more than acquisition strategy, while larger LMs remain future work.

5 Conclusion

  • The proposed approach combines task-adaptive pretraining on the unlabeled pool with fine-tuning robust to low- and high-resource settings.This uses the entire unlabeled pool alongside labeled data during active learning.Source: §5 · p. 5
  • Experiments on five standard NLP datasets show substantially better results than standard fine-tuning.Source: §5 · p. 5
  • Training strategy can be more important than acquisition strategy, and poor training can limit even a good acquisition function.Source: §5 · p. 5
  • The authors identify larger language models such as GPT-3 and FLAN as future candidates for challenging low-resource active learning settings.Source: §5 · p. 5

A.1 Datasets

The evaluation covers five diverse natural language understanding tasks spanning question classification, sentiment analysis, and topic classification. The datasets vary in label structure and scale, with task-specific validation and held-out test-set procedures.

A.1 Datasets

  • The study evaluates five tasks: TREC-6 question classification, IMDB and SST-2 sentiment analysis, and AGNEWS and DBPEDIA topic classification.The tasks include binary and multi-class labels and varying dataset sizes.Source: §A.1 · p. 9
  • AGNEWS and DBPEDIA are large-scale datasets, with DBPEDIA undersampled to 20K pool examples and 2K validation examples for active-learning simulation.Source: §A.1 · p. 9
  • Validation data are sampled at 10% for TREC-6, IMDB, and SST-2, 5% for AGNEWS, and from standard splits with undersampling for DBPEDIA.Source: §A.1 · p. 9
  • Standard test sets are used except for SST-2, where the GLUE development set serves as the held-out test set.Source: §A.1 · p. 9

A.2 Training & AL Details

The experiments use BERT-based models, multiple acquisition functions, and two fine-tuning strategies under standardized training and evaluation settings. Baselines include random, uncertainty-, diversity-, and hybrid acquisition methods, with results measured across repeated runs.

A.2 Training & AL Details

  • TAPT fine-tunes BERT-BASE for 100K steps with learning rate 2e −05, evaluating five times per epoch and selecting the model with lowest validation loss.Source: §A.2 · p. 9
  • All datasets use a 15% Dpool annotation budget, five random seeds, held-out testing, and a single Nvidia Tesla V100 GPU per experiment.Source: §A.2 · p. 9

A.3 Hyperparameters

  • BERT training uses batch size 16, learning rate 2e −5, AdamW with epsilon 1e −8, no weight decay, and maximum sequence lengths of 128 or 256.Source: §A.3 · p. 9

A.4 Baselines

  • Acquisition baselines include RANDOM, BADGE, ALPS, and BERTKM alongside an entropy-based method.BADGE combines diversity and uncertainty, ALPS uses masked-language-model surprisal embeddings, and BERTKM clusters normalized BERT embeddings.Source: §A.4 · p. 9
  • Experiments compare original BERT and adapted BERT-TAPT with proposed FT+ and standard SFT fine-tuning.Source: §A.4 · pp. 9–10
  • Table 2 reports validation and test accuracy for BERT and BERT-TAPT across datasets, averaged over runs with different random seeds.Source: Table 2 · p. 10

B.1 Task-Adaptive Pretraining (TAPT) & Full-Dataset Performance

The section evaluates task-adaptive pretraining, fine-tuning, acquisition functions, and their computational costs in active learning. It finds that training strategy substantially influences performance, while acquisition methods differ in efficiency and often yield similar results.

B.1 Task-Adaptive Pretraining (TAPT) & Full-Dataset Performance

  • The study evaluates TAPT by continuing BERT pretraining on the unlabeled pool, selecting checkpoints with the lowest validation loss across learning rates.Learning curves show that a smaller learning rate facilitates masked-language-model training, and the resulting model is used in TAPT experiments.Source: §B.1 · p. 10, Figure 5 · p. 10
  • Table 2 reports validation and test accuracy for BERT and BERT-TAPT, using the proposed FT+ method and means across three random seeds.The supplied passage identifies the compared models and aggregation procedure but does not provide the table’s accuracy values.Source: §B.1 · p. 10

B.2 Performance of Acquisition Functions

  • The acquisition-function comparison includes RANDOM, ALPS, BADGE, BERTKM, ENTROPY, LEAST CONFIDENCE, BALD, and BATCHBALD.BALD uses mutual information between predictions and posterior probabilities, whereas BATCHBALD jointly scores batches through mutual information with model parameters.Source: §B.2 · pp. 10–11, §B.3 · p. 11
  • Performance curves are generally close, suggesting that training strategy can matter more than acquisition strategy in some cases; ENTROPY is generally best, followed by BADGE.The authors also report that BALD slightly underperforms while the other uncertainty-based functions perform similarly.Source: §B.2 · p. 11
  • All acquisition strategies benefit from combining TAPT with FT+, while uncertainty-based acquisition generally outperforms diversity-based methods.The comparison covers AGNEWS and IMDB using BERT-TAPT-FT+; ENTROPY is generally strongest among uncertainty methods.Source: §B.2 · p. 11

B.3 Efficiency of Acquisition Functions

  • Runtime analysis separates inference time from selection time: ALPS and BERTKM are faster, BADGE and BATCHBALD are most expensive, and RANDOM takes zero seconds.Uncertainty methods may require inference over the pool, while iterative selection algorithms increase selection time.Source: §B.3 · pp. 11–12, Table 3 · p. 11

Related papers

Loading 2104.08320v2…