Source-linked AI summary
Learning how to Active Learn: A Deep Reinforcement Learning Approach
Meng Fang, Yuan Li, Trevor Cohn
TL;DR
Annotated data is costly, while fixed active-learning heuristics have limited and dataset-dependent effectiveness. The paper reframes active learning as reinforcement learning, learns a data-selection policy, and transfers it across languages. It reports consistent and sizeable improvements over baseline methods, including traditional uncertainty sampling, with annotation burden reduced to as low as 10% versus random sampling.
Problem
Fixed active-learning heuristics have limited effectiveness, vary across datasets, and are difficult to learn or apply in low-resource languages.
Method
PAL formulates active learning as a Markov decision process and learns a dynamic data-selection policy with deep reinforcement learning for transfer between languages.
Results
The learned policies provide consistent and sizeable improvements over baseline methods, including traditional uncertainty sampling, while PAL methods reduce annotation burden to as low as 10% versus random sampling.
Takeaways & Limitations
Policy learning can support cross-lingual active learning, including a cold-start setting without evaluation data or the ability to react to annotations.
Takeaways & Limitations
The transfer algorithm requires held-out evaluation data and embeds the oracle annotator in the learning loop, limiting realism for low-resource field-linguistics settings.
Abstract
from arXiv · showhide
Active learning aims to select a small subset of data for annotation such that a classifier learned on the data is highly accurate. This is usually done using heuristic selection methods, however the effectiveness of such methods is limited and moreover, the performance of heuristics varies between datasets. To address these shortcomings, we introduce a novel formulation by reframing the active learning as a reinforcement learning problem and explicitly learning a data selection policy, where the policy takes the role of the active learning heuristic. Importantly, our method allows the selection policy learned using simulation on one language to be transferred to other languages. We demonstrate our method using cross-lingual named entity recognition, observing uniform improvements over traditional active learning.
1 Introduction
Active learning reduces annotation costs by selecting data for supervised learning, but existing heuristics are difficult to learn and transfer across languages. PAL addresses this by learning a dynamic reinforcement-learning policy that can transfer between languages.
- Annotated text is a critical bottleneck for training accurate NLP models, motivating active learning to reduce annotation expense.
- Prior active-learning methods typically rely on fixed heuristics, while comparatively little work learns the selection strategy itself.
- PAL formulates active learning as reinforcement learning, where an agent makes binary annotation decisions in a stream using sentence content, model classifications, and confidence.
- PAL learns a policy on a high-resource language and transfers it to another language using cross-lingual word embeddings for compatible representations.
2 Related work
Related work frames active learning as selecting informative data through uncertainty, disagreement, or expected error reduction, while deep reinforcement learning provides a broader decision-making framework. Prior research has not substantially addressed learning transferable active-learning strategies across languages.
- Existing active-learning methods include uncertainty sampling, query by committee, and expected error reduction.
- Active learning has been applied across NLP tasks, including text classification, relation classification, and structured prediction.
- Deep reinforcement learning supports decision making through representation learning and has been studied for NLP information extraction.
- Prior policy-transfer research did not address transferring active-learning strategies between languages with a shared feature space.
3 Methodology
The methodology reframes active learning as a decision process so that deep reinforcement learning can learn the selection policy from data and transfer it between languages.
- The method formalises active learning as a decision process and learns its selection policy from data using deep reinforcement learning.
- The methodology later introduces a procedure for transferring the learned policy between languages.
3.1 Active learning as a decision process
Active learning can be represented as sequential decisions over labelled and unlabelled data, with a heuristic determining which instances to annotate. The resulting Markov decision process supports dynamic policy learning under a fixed annotation budget.
- Active learning repeatedly selects unlabelled instances for human annotation until a termination criterion, such as exhausting the budget, is reached.
- At each stage, the system state contains selected labelled data and labels, while a heuristic chooses the next items from an unlabelled pool.
- The heuristic acts as a policy mapping the current labelled-data state and candidate information, such as model uncertainty, to a labelling decision.
3.2 Stream-based learning
The paper formulates streaming active learning as an MDP in which an agent sequentially decides which instances to annotate under a fixed budget. It represents each state using the candidate sentence and model outputs, then learns a selection policy with deep Q-learning using held-out performance rewards.
- 3.2 Stream-based learning: In the streaming setup, each arriving sentence triggers a binary decision to annotate or skip it, with the process ending when data or the annotation budget is exhausted.The model is updated when an instance is labelled, and the final model is trained on the selected dataset.
- 3.2 Stream-based learning: The active-learning process is represented as an MDP whose state includes the current sentence, labelled data, predictive model, actions, rewards, and transition dynamics.The action set is A = {0, 1}, denoting whether to label the current instance.
- 3.2.1 State: Each state combines a fixed-dimensional sentence representation with predictive marginal distributions and a confidence representation from the trained model.A convolutional network encodes sentence content, while another processes predictive marginals to capture patterns across adjacent positions.
- 3.2.3 Reward: The policy receives intermediate rewards equal to changes in held-out predictive performance, with total reward measuring performance improvement over the episode.Here, predictive accuracy is measured using F1 score, and intermediate rewards may be positive or negative.
- 3.2.5 Reinforcement learning: Deep Q-learning learns the policy by estimating action utility from episode transitions and updating parameters with replayed minibatches and gradient descent.The algorithm constructs states, selects actions, receives held-out-set rewards, stores transitions, and minimizes the Q-learning loss.
3.3 Cross-lingual policy transfer
The learned active-learning policy is transferred from a source language to a different target language in a cross-lingual NER setting. Cross-lingual word embeddings provide shared representations, while the transferred policy is adapted during a single target-language pass.
- 3.3 Cross-lingual policy transfer: Policy transfer trains an active-learning policy on a source language and applies it to a different target language for the same NER task.The approach is intended for a different data setting, such as a different domain, task, or language.
- 3.3 Cross-lingual policy transfer: Cross-lingual word embeddings create compatible representations that allow the learned policy to be ported between languages.The paper highlights English as a high-resource source language where policy learning is easier.
- 3.3 Cross-lingual policy transfer: The transfer algorithm makes one pass over the target-language data and fine-tunes an initial policy using held-out performance.The single-pass design suits low-resource settings where oracle labelling is costly.
3.4 Cold-start transfer
The cold-start transfer setting addresses data- and communication-impoverished conditions by making one batch selection without held-out feedback or policy updates. It transfers both a policy and a pretrained model across languages to support target-language selection.
- Limitations: The transfer algorithm has limitations: it requires held-out evaluation data and embeds the oracle annotator inside the learning loop.Held-out evaluation adds supervision, while oracle integration constrains annotator communication and requires real-time processing.
- Cold-start constraints: Cold-start permits only one target-data labeling request and forbids policy updates because no held-out data is available.The agent selects a batch without using the resulting annotations or other feedback to refine selection.
- Policy and model transfer: Cold-start transfers a policy and model into the target language using cross-lingual word embeddings.The model is trained on one source language, while policy learning uses a different source language and initializes the model with the transferred model.
4 Experiments
Experiments evaluate policy transfer for cross-lingual named entity recognition in bilingual, multilingual, and cold-start settings. PAL consistently outperforms the baselines, with multilingual training and pretrained models providing particularly strong results.
- Experimental setup: The experiments transfer an active learning policy from source languages to target languages, using repeated source simulations but one target-language episode.The evaluation includes bilingual, multilingual, and cold-start configurations across English, German, Spanish, and Dutch NER corpora.
- Experimental setup: Figure 3 compares bilingual and multilingual policy learning across three target languages.Bilingual policies train only on English, whereas multilingual policies train on all other languages excluding the target.
- Bilingual results: PALb consistently outperforms Random and Uncertainty baselines across the three target languages.Content information helps PALb start more strongly despite a poor initial model, while uncertainty sampling depends on a high-quality model.
- Multilingual results: PALm outperforms all other approaches, including PALb, with especially marked early-stage gains for Spanish and Dutch.The results indicate that joint training over several languages produces a better policy than training on one source language.
- Cold-start results: PALc outperforms both baselines in cold-start evaluation, where policy and model updates are unavailable and annotations arrive in a batch.All methods start near 40% because the model is initialized with an NER tagger trained on another language.
- Overall results: The PAL methods all outperform the baselines, and annotation costs fall to as low as 10% of random sampling while matching Random performance.Table 2 reports F1 scores and relative cost reductions for training sets of 200 labeled sentences; PALm exceeds PALb, while PALc achieves the best overall results.
5 Conclusion
The paper formulates active learning as sequential binary annotation decisions in a Markov decision framework and learns the selection policy with deep reinforcement learning. Learned policies transfer between languages and improve over baseline methods, including uncertainty sampling, even in cold-start conditions.
- Contribution: The proposed algorithm learns active learning strategies from data rather than relying on a fixed selection heuristic.Active learning is represented as a sequence of binary annotation decisions over a stream of data.
- Contribution: The selection policy is designed as a deep reinforcement learning policy within a Markov decision framework.The formulation treats annotation decisions as sequential actions in the active learning process.
- Results: Learned active learning policies transfer between languages and provide consistent and sizeable improvements over baseline methods, including uncertainty sampling.The conclusion states that this improvement also holds in a difficult cold-start setting without evaluation data or reaction to annotations.