Source-linked AI summary
Deep Active Learning for Named Entity Recognition
Yanyao Shen, Hyokun Yun, Zachary C. Lipton, Yakov Kronrod, Animashree Anandkumar
TL;DR
Deep NER models typically need substantial labeled data, even though NER annotation is costly and unlabeled text is abundant. The paper combines incremental active learning with a lightweight CNN-CNN-LSTM architecture and reports near-state-of-the-art performance using much less data. On OntoNotes, it reaches 99% of the best deep-model performance with 24.9% of English data and 30.1% of Chinese data.
Problem
Deep learning’s advantage for NER diminishes on smaller labeled datasets, while obtaining annotated NER data is expensive.
Method
The paper uses incremental active learning with a CNN-CNN-LSTM NER architecture and uncertainty-based sample selection.
Results
99% of the best deep-model performance is achieved using 24.9% of OntoNotes-5.0 English data and 30.1% of OntoNotes-5.0 Chinese data.
Takeaways & Limitations
Deep active learning achieves state-of-the-art performance with substantially fewer labeled samples than standard supervised training.
Takeaways & Limitations
The annotation setup assumes that sentence-labeling cost is proportional to sentence length in words.
Abstract
from arXiv · showhide
Deep learning has yielded state-of-the-art performance on many natural language processing tasks including named entity recognition (NER). However, this typically requires large amounts of labeled data. In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning. While active learning is sample-efficient, it can be computationally expensive since it requires iterative retraining. To speed this up, we introduce a lightweight architecture for NER, viz., the CNN-CNN-LSTM model consisting of convolutional character and word encoders and a long short term memory (LSTM) tag decoder. The model achieves nearly state-of-the-art performance on standard datasets for the task while being computationally much more efficient than best performing models. We carry out incremental active learning, during the training process, and are able to nearly match state-of-the-art performance with just 25\% of the original training data.
1 INTRODUCTION
Deep learning’s advantage for NER diminishes on smaller labeled datasets, motivating active learning to select informative examples under costly annotation. The paper combines incremental training with a lightweight CNN-CNN-LSTM architecture and reports near-best performance using substantially less data.
- Motivation: On CoNLL-2003 English, the best DNN model’s advantage over the best shallow model is only 0.4% F1, compared with 2.24% on OntoNotes-5.0 English.The datasets contain 203,621 and 1,088,503 training words, respectively.
- Motivation: Active learning selects which examples to annotate instead of labeling randomly drawn examples, addressing expensive NER annotation.Unlabeled NER data can be scraped efficiently, whereas annotation requires costly multi-stage pipelines and trained annotators.
- Approach: Incremental training mixes newly annotated samples with older ones and updates weights for a few epochs, avoiding complete retraining after every round.Traditional active learning required 54 retrainings in the authors’ experiments.
- Approach: CNN-CNN-LSTM combines convolutional character and word encoders with an LSTM tag decoder to reduce computational complexity while performing competitively.The architecture also handles out-of-vocabulary words gracefully and relies more on convolutions than recurrent layers.
- Results: 99% of the best deep-model F1 is achieved with 24.9% of OntoNotes-5.0 English data and 30.1% of OntoNotes-5.0 Chinese data.The reported results use the paper’s uncertainty-based active-learning approach.
2 RELATED WORK
Prior NER research developed neural architectures and practical active-learning heuristics, but deep active learning for sequence tagging remained largely unstudied. Sequence tagging presents structured outputs and variable-length inputs that distinguish it from much of the earlier image-classification work.
- Deep learning for NER: Earlier NER architectures progressed from temporal CNNs to bidirectional LSTM encoders and hierarchical character-aware models.
- Active learning: Active-learning research includes margin-based, multi-criterion, and least-confidence selection procedures, but classic guarantees do not generalize to DNNs for NER.
- Deep active learning: Most deep active-learning work addressed image classification, while sequence tagging involves structured output spaces and variable-length input.
- Deep learning for NER: Table 1 organizes prior sequence-tagging architectures by their corresponding design choices.The paper’s framework separates character-level encoding, word-level encoding, and tag decoding.
3 NER MODEL DESCRIPTION
The model is organized as character-level and word-level encoders followed by a tag decoder, with CNN-based encoding and an LSTM decoder chosen to support efficient NER training. This lightweight design targets computationally efficient retraining while retaining competitive performance.
- Model components: NER architectures are decomposed into character-level encoding, word-level encoding, and tag decoding components.The character encoder extracts word features from characters, the word encoder uses surrounding words, and the tag decoder models tag-sequence probabilities.
- Model components: CNN-CNN-LSTM combines convolutional character and word encoders with an LSTM tag decoder.The architecture is proposed as a lightweight NER network because CNNs are computationally more efficient than LSTMs for encoding.
- Character-level encoder: CNN character encoding is preferred because its slight performance disadvantage relative to LSTM encoding is not statistically significant, while its computational cost is much lower.Character features are extracted with CNNs and converted to fixed-length word representations using max-pooling.
- Word-level encoder: The word-level encoder uses CNNs to extract representations from word-level input features for each sentence position.The illustrated architecture uses two convolutional layers with kernels of width 3.
- Tag decoder: The LSTM tag decoder predicts tags sequentially, feeding each decoded tag into the next time step.It starts with a [GO] symbol and uses a softmax loss for decoding.
- Tag decoder: The locally normalized LSTM decoder avoids CRF partition-function computation and significantly speeds training while achieving performance comparable to a chain CRF decoder.Greedy left-to-right decoding is used despite exact best-sequence decoding being computationally intractable.
4 ACTIVE LEARNING
The active-learning procedure selects informative unlabeled NER sentences using uncertainty-based rankings and trains incrementally as new labels arrive. It also evaluates Bayesian disagreement and representativeness-based alternatives, with the latter failing to improve on uncertainty heuristics in the reported experiments.
- Setup: Active learning repeatedly selects sentences for annotation from unlabeled data, addressing the high cost of manual NER annotation.The setup acquires annotations over multiple rounds.
- Uncertainty sampling: Uncertainty sampling ranks unlabeled examples by the model’s confidence in their predicted tag sequences.The study considers three ranking methods that can be implemented in CNN-CNN-LSTM and related NER models.
- Uncertainty sampling: Least Confidence scores examples by one minus the probability of the most likely tag sequence, approximated using the greedily decoded sequence.Exact computation would require finding the most likely LSTM-decoded tag sequence, which is intractable.
- Uncertainty sampling: Maximum Normalized Log-Probability normalizes sequence log probability by sentence length because Least Confidence disproportionately selects longer sentences.The normalization also reflects that longer sentences require more annotation labor.
- Alternative sampling: BALD measures uncertainty from prediction variability across dropout-based forward passes.The experiments draw 100 independent dropout masks for this method.
- Alternative sampling: Representativeness-based sampling weights marginal utility over unlabeled points by uncertainty and uses a submodular online algorithm under a knapsack constraint.In the experiments, it fails to match uncertainty heuristics or improve them in combination.
5 EXPERIMENTS
Experiments evaluate CNN-CNN-LSTM and active-learning strategies on standard NER datasets, finding strong performance and substantially reduced labeling requirements. The lightweight model also improves training efficiency relative to recurrent and CRF-based alternatives.
- Model efficiency and performance: CNN-CNN-LSTM trains twice as fast as CNN-CNN-CRF on OntoNotes 5.0 English.The speed advantage is attributed to CRF partition-function complexity, which is quadratic in the number of entity types.
- Model efficiency and performance: CNN-CNN-LSTM is 44% faster than CNN-LSTM-LSTM on OntoNotes and provides four times the training-speed improvement over CNN-LSTM-CRF.The comparison supports the efficiency of convolutional encoders and the lightweight decoder design.
- Active-learning performance: All active-learning algorithms significantly outperform the random baseline, with MNLP and BALD slightly ahead of LC in early rounds.MNLP is faster to compute than BALD because it requires one forward pass rather than multiple passes.
- Active-learning performance: Active learning surpasses full-data shallow-model performance with 12.0% of English training data and 16.9% of Chinese training data.The experiment began with 1% of the original training data and added 20,000 words per round.
- Genre coverage: Training on unbiased half-data achieves 85.10 F1, exceeding no-nw-data at 81.49 and nw-only-data at 82.08.The genre experiment shows that active learning selected under-explored genres despite receiving no genre labels.
6 CONCLUSION
The paper proposes an efficient NER model and demonstrates that deep active learning can achieve state-of-the-art performance with substantially less labeled data.
- Conclusion: The proposed efficient NER model achieves high performance on well-established datasets.The conclusion presents model efficiency and performance as central outcomes.
- Conclusion: Deep active learning achieves state-of-the-art performance with much less data than standard supervised training.This conclusion summarizes the paper’s empirical finding across its NER experiments.
A EFFECT OF BEAM SIZE ON LSTM DECODER
The appendix examines how beam size affects LSTM-decoder performance and finds that greedy decoding is effective despite exact best-sequence decoding being computationally intractable.
- Decoding complexity: Finding the best label sequence for an LSTM decoder is computationally intractable.The appendix motivates evaluating simpler decoding strategies instead.
- Beam-size results: Greedy decoding with beam size 1 performs surprisingly well, while beam size 2 is only marginally better.Further increases in beam size do not improve performance appreciably.
- Beam-size results: Table 5 evaluates beam-size effects using an LSTM-LSTM-LSTM model on OntoNotes 5.0 English.The table compares decoder performance across beam sizes.
B LEARNING CURVE IN ACTIVE LEARNING EXPERIMENTS ACROSS MULTIPLE RUNS
Repeating the English active-learning experiment eight times with different random initial datasets shows consistent differences among selection methods. MNLP and BALD tend to outperform LC in early acquisition rounds.
- Experimental repetition: The active-learning experiment was repeated eight times with different randomly selected initial datasets.The repetitions were designed to assess variability in learning curves across experiments.
- Learning-curve comparison: MNLP, LC, and BALD are competitive overall, while MNLP and BALD outperform LC in early acquisition rounds.This trend was observed during the first nine rounds of labeled-data acquisition.
C REPRESENTATIVENESS-BASED ACTIVE LEARNING
Representativeness-based active learning selects informative, diverse NER sentences under a labeling budget, combining uncertainty filtering with submodular optimization and streaming selection.
- Sample representation: Samples are embedded as fixed-dimensional vectors using average pretrained word embeddings or average top-layer encoder activation maps.Pairwise similarity can be defined through L1/L2 distance or cosine similarity.
- Utility function: The labeling utility measures incremental similarity gain between selected samples, labeled data, and the remaining data, forming a monotone submodular knapsack problem.Knapsack constraints account for different sentence lengths and therefore different labeling costs.
- Candidate selection: Uncertainty sampling narrows the unlabeled pool before representativeness-based selection, using the top t · K samples with t = 4 in experiments.This restriction improves computational efficiency while retaining candidates prioritized by model uncertainty.
- Optimization: A two-pass streaming algorithm replaces offline greedy optimization for computationally efficient sample selection under the knapsack constraint.Its stated complexity is Õ(ln)^3 when logarithmic factors are ignored.
- Optimization: The streaming procedure maintains candidate sets across value buckets and updates them using marginal submodular improvement until the budget or improvement criterion stops further updates.The resulting approximation guarantee depends on δ, which was around 0.01 in the experiments and is described as negligible in practical acquisition settings.