Source-linked AI summary
Few-shot classification in Named Entity Recognition Task
Alexander Fritzler, Varvara Logacheva, Maksim Kretov
TL;DR
NER often lacks enough labelled data, especially for rare entities and low-resource settings. The paper adapts prototypical networks, a metric-learning approach, to semi-supervised few-shot NER using information from other classes. It reports better few-shot performance than an RNN baseline, while zero-shot NER remains unsuccessful and multi-class scaling is left for future work.
Problem
NER data is often insufficient for rare entities and low-resource settings, creating a need for methods that learn from very small labelled sets.
Method
The paper adapts prototypical networks to semi-supervised NER, using limited target-class data together with labelled examples from other classes.
Results
Prototypical networks outperform the state-of-the-art RNN model in few-shot NER, although they do not succeed in zero-shot NER.
Takeaways & Limitations
The approach is suitable for NER settings where extremely little target-class data is available and can in principle support zero-shot text classification.
Takeaways & Limitations
The experiments predict entities for only a single class, leaving performance on multiple classes for future work.
Abstract
from arXiv · showhide
For many natural language processing (NLP) tasks the amount of annotated data is limited. This urges a need to apply semi-supervised learning techniques, such as transfer learning or meta-learning. In this work we tackle Named Entity Recognition (NER) task using Prototypical Network - a metric learning technique. It learns intermediate representations of words which cluster well into named entity classes. This property of the model allows classifying words with extremely limited number of training examples, and can potentially be used as a zero-shot learning method. By coupling this technique with transfer learning we achieve well-performing classifiers trained on only 20 instances of a target class.
1 INTRODUCTION
NER identifies entity names in text but is difficult to train for rare classes and low-resource settings. The paper adapts prototypical networks to few-shot NER and reports stronger low-resource performance than a state-of-the-art model.
- NER identifies person, organization, and location names in unstructured text, often treating the task as sequence labelling.
- Large labelled datasets support high-quality NER, but abundant data is often unavailable for low-resourced languages and rare entities.
- Few-shot NER seeks to identify entities using extremely small numbers of labelled examples, motivating semi-supervised methods that exploit unlabelled data.
- Metric learning uses class structure to label uncommon instances from few examples and can potentially support zero-shot classification without retraining for new classes.
- The paper applies prototypical networks to NER and compares them with commonly used baselines in low-resource settings.
- Prototypical networks do not succeed in zero-shot NER but outperform other models in the few-shot case.
2 RELATED WORK
Prior low-resource NER work uses transfer across domains or languages, external information, active or self-learning, and specialized embedding methods. This paper positions prototypical networks as a less-used NLP approach with potential zero-shot label introduction.
- Low-resource NER research commonly uses transfer learning because neural architectures are sensitive to training-set size and can overfit small datasets.
- Researchers transfer NER information across domains or languages by transforming features, mapping entities, or translating datasets.
- Ontologies, knowledge bases, heuristics, active learning, and self-learning provide additional strategies for limited-data NER.
- Related metric-based work learns embeddings and selects prototypes for fine-grained hierarchical NER, whereas this paper does not restrict classes to a hierarchy.
- Metric learning and meta-learning exploit shared structure or cross-task regularities to learn from few examples.
- The paper states that prototypical networks had not previously been applied to NLP and offer the possibility of introducing new labels without retraining.
3 PROTOTYPICAL NETWORKS
Prototypical networks map inputs into a task-specific space, average support representations into class prototypes, and classify queries by prototype similarity. For NER, the paper preserves sentence structure, treats the non-entity class specially, and uses in-domain examples alongside out-of-domain data.
- The network learns representations that cluster same-class objects and separate different-class objects, enabling classification of unseen objects from scarce labels.
- Training samples a labelled support set and a labelled query set, randomly repartitioned from the training data at each step.
- For each class, the model averages support representations into a prototype in an M-dimensional space.
- A query is mapped into the same space and assigned to the class with the closest prototype, with similarities converted into class probabilities by softmax.
- NER support and query sets use whole sentences rather than individual words to avoid breaking sentence structure.
- The model replaces the similarity score for the non-entity class O with a scalar because O words should not cluster together.
- The adapted few-shot setting trains with a small labelled in-domain target-class set and larger out-of-domain data for other classes.
4 FEW-SHOT NER
The paper adapts prototypical networks to few-shot NER, using small target-class data alongside out-of-domain examples. It compares baseline, transfer-learning, and CRF-enhanced variants for identifying rare entities.
- Task formulation: Few-shot NER targets rare entity classes when only a very limited number of labelled examples is available.The setup treats each target class separately and frames the task as a cold-start problem for incorporating new labels.
- Prototypical network: The adapted prototypical network maps RNN hidden states into an embedding space, constructs prototypes from support examples, and classifies query words by prototype similarity.The model can be used with or without a CRF layer.
- Protonet training: Training alternates between target-class in-domain data and randomly selected out-of-domain classes, exposing the representation to multiple classes.This regularised Protonet procedure is intended to make the prototype-space mapping more robust while leveraging out-of-domain data.
- Transfer learning: WarmBase pre-trains an RNN baseline on out-of-domain data before retraining its non-CRF weights on the in-domain target class.This provides a transfer-learning comparison for the prototypical-network models.
- Transfer learning: WarmProto initializes Protonet with a Base model trained on out-of-domain data, while WarmProto-CRF adds a CRF layer on top of the pretrained prototypical network.WarmProtoZero removes target-class exposure during training and uses target examples only as a test-time support set.
5 EXPERIMENTAL SETUP
Experiments use Ontonotes to simulate few-shot target-class learning with carefully controlled sampling and shared hyperparameters. Models are evaluated across all 18 entity classes using chunk-based F1-score.
- Dataset: Ontonotes contains 18 entity classes plus O, with over 30,000 training instances for some common classes and fewer than 100 for rare classes.The dataset contains 150,374 training sentences and 19,206 validation sentences.
- Dataset: The experiments preserve BIO labelling, where B and I distinguish entity beginnings from continuation tokens.The O label denotes tokens outside the desired entities.
- Data preparation: For each target class C, training labels retain C in the in-domain validation data and remove C from the out-of-domain training data.All other labels are converted to O in the in-domain data, while B-C and I-C are converted to O out of domain.
- Data preparation: Sampling adds class-free sentences so the target-class frequency matches the validation distribution and reduces variation in in-domain dataset size.The procedure samples N sentences containing C and N × (1−pr(C)) sentences without C.
- Evaluation: Results are averaged across four random-seed experiments for each of the 18 classes, using chunk-based F1-score as the evaluation metric.Hyperparameters are tuned on three well-represented classes and then reused for other classes; Table 1 reports means with standard deviations.
- Design of experiments: 20 target-class instances are used because they can be labelled by hand while producing models of reasonable quality.Figure 3 compares models trained on 10 and 20 examples, with the larger set producing a significant performance boost.
6 RESULTS
The results show that out-of-domain information improves few-shot NER, especially for prototypical networks combined with transfer learning and a CRF layer. Zero-shot training, however, performs poorly, and BIO removal does not materially change most scores.
- Performance of models: 20 sentences are insufficient to train a reliable baseline RNN+CRF model, while BaseProto performs closely to the RNN baseline.The results also indicate that 20 target-class instances alone are insufficient to construct a reliable prototype.
- Performance of models: Protonet performs better than Base and BaseProto on many classes after exposure to instances of other classes.The comparison supports using other-class structure during few-shot prototype learning.
- Transfer learning: WarmBase achieves results comparable with Protonet, indicating that out-of-domain pre-training is useful for the conventional RNN baseline.The passage also states that information about other classes helps the baseline.
- Transfer learning: WarmProto beats WarmBase and Protonet in more than half of experiments, but WarmProtoZero performs below every other tested model.WarmProto benefits from out-of-domain pre-training, whereas the zero-shot variant receives no target-class instances during training.
- Performance of models: WarmProto-CRF beats all other models in almost all experiments after adding a CRF layer to the pretrained prototypical network.The authors conclude that prototypical networks are more effective than the RNN baseline when in-domain data is extremely limited.
- BIO labelling: Removing BIO tags produces no significant F1-score differences for WarmBase and WarmProto in most cases.The experiment replaces B-C and I-C with C while leaving the chunk-based F1 evaluation unchanged.
7 CONCLUSIONS
The paper adapts prototypical networks to NER, using class prototypes and semi-supervised information to handle very limited labelled data. It outperforms an RNN baseline in this setting, but zero-shot NER remains imperfect and performance varies across entity classes.
- Prototypical networks project objects into a vector space where same-class representations form prototypes used to classify new objects by similarity.The method was originally developed for image classification and is adapted here to text analysis.
- The model uses information from words of other classes to identify target classes in a semi-supervised NER setting.This setting is intended for cases with an extremely small amount of labelled data.
- The prototypical network is more effective than the state-of-the-art RNN model in the semi-supervised setting.
- Zero-shot NER is possible in principle with prototypical networks, although the authors report substantial room for improvement.The authors identify zero-shot text classification as a direction for future research.
- Performance differs considerably across named-entity classes, motivating qualitative analysis of which textual characteristics suit the method.
- Current experiments predict entities from only one class, while future work will test whether performance scales to multiple classes.The planned extension targets predicting all classes in Ontonotes or another NER dataset at once.