Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

Yi Yang; Arzoo Katiyar

Source-linked AI summary

Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

Yi Yang, Arzoo Katiyar

arXiv:2010.02405v1Oct 6, 2020cs.CL

TL;DR

Few-shot NER must transfer across domains with limited labels and mismatched entity types while modeling sequence-level label dependencies. The paper uses source-domain supervised NER representations for nearest-neighbor classification and adds Viterbi decoding. STRUCTSHOT achieves state-of-the-art results, outperforming prior systems by 6%–16% F1 across one- and five-shot settings.

Problem
Few-shot NER requires adapting to new domains with few labels, differing entity types, and structured label dependencies that prototype-based methods handle poorly.
Method
STRUCTSHOT uses a source-domain supervised NER model as a token-level feature extractor, then combines nearest-neighbor classification with Viterbi decoding.
Results
STRUCTSHOT achieves new state-of-the-art results, outperforming the previous state-of-the-art system by 6%–9% F1 in one-shot and 11%–16% F1 in five-shot settings.
Takeaways & Limitations
Nearest-neighbor learning with structured decoding provides an effective few-shot NER system without few-shot-specific training.
Takeaways & Limitations
STRUCTSHOT struggles to distinguish highly ambiguous entity classes, especially numerical types, without domain-specific knowledge.

Abstract

from arXiv · show

We present a simple few-shot named entity recognition (NER) system based on nearest neighbor learning and structured inference. Our system uses a supervised NER model trained on the source domain, as a feature extractor. Across several test domains, we show that a nearest neighbor classifier in this feature-space is far more effective than the standard meta-learning approaches. We further propose a cheap but effective method to capture the label dependencies between entity tags without expensive CRF training. We show that our method of combining structured decoding with nearest neighbor learning achieves state-of-the-art performance on standard few-shot NER evaluation tasks, improving F1 scores by $6\%$ to $16\%$ absolute points over prior meta-learning based systems.

1 Introduction

Few-shot NER must adapt across domains with mismatched entity types and limited annotations. STRUCTSHOT addresses these challenges by using contextual token representations, nearest-neighbor classification, and structured decoding.

Cross-domain NER adaptation is difficult because target domains can use entity types that differ substantially from source annotations, while new-domain labeling is expensive.Only two of I2B2 2014’s twenty-three entity types appear in OntoNotes 5.
Prior few-shot NER methods based on prototypical networks achieve only 10∼30% F1 on average for one- or five-shot cross-dataset transfer.
NER is structured rather than independent token classification, and the heterogeneous Outside class makes a single prototype noisy.Professions and dates can both belong to O, despite differing semantics.
STRUCTSHOT represents support tokens with contextual representations from a source-domain supervised NER model instead of learning class prototypes.
STRUCTSHOT combines nearest-neighbor classification with Viterbi decoding, avoiding few-shot-specific training while modeling label dependencies.For each target token, nearest-neighbor matching retrieves a similar support token rather than relying on an erroneous class-level O representation.

2 Problem Statement and Setup

Few-shot NER evaluates sequence labeling on target domains using only a few labeled examples per entity class, even when target tag sets differ from source domains. The proposed setup samples support sets but evaluates on standard benchmark test sets for more realistic and reproducible comparisons.

2.1 Few-shot NER: NER assigns each token an entity label or O, with BIO or IO schemes indicating whether tokens begin or continue entities.
2.1 Few-shot NER: Few-shot NER tests models on target domains with only a few labeled examples per entity class, potentially requiring generalization to unseen entity classes.
2.1 Few-shot NER: The K-shot task supplies a sentence and a support set containing K entity examples for each target entity class, then seeks the best tag sequence.
2.2 A standard evaluation setup: Prior episode evaluation samples both support and test sets, whereas the proposed setup samples support sets and evaluates on standard benchmark test sets.
2.2 A standard evaluation setup: Episode evaluation with equally distributed test classes does not reflect real entity distributions or model effectiveness when adapting to a new domain.For example, DATE appears 4,983 times in I2B2 test data, while EMAIL appears once.
2.2 A standard evaluation setup: Support sets are greedily sampled from development data, with multiple samples averaged on the test set to account for performance variance and support reproducibility.

3 Model

STRUCTSHOT combines source-domain NER representations with token-level nearest-neighbor classification and Viterbi decoding to handle target-domain label dependencies without CRF training.

Nearest neighbor classification: STRUCTSHOT uses a source-domain NER model as a token embedder, then classifies target tokens by nearest-neighbor matching against contextualized support tokens.The approach can use BiLSTM or BERT-based NER encoders and does not simulate few-shot episodes during training.
Nearest neighbor classification: Nearest-neighbor classification assigns each test token the tag of the most similar support token using squared Euclidean distance over L2-normalized features.The support tokens are grouped by tag, and similarity is computed in the learned contextual representation space.
Structured decoding: The abstract transition matrix distributes each abstract transition probability evenly across related target transitions, preserving a cheap mechanism for structured inference.The extension maps same-colored cells in corresponding matrix rows to related target transitions.
Structured decoding: STRUCTSHOT captures label dependencies with abstract O, I, and I-Other transitions estimated from source-domain counts, then extends them to target-specific transitions.The method avoids learning target-domain CRF transition scores, which are difficult to transfer because source and target tag sets differ.
Structured decoding: Viterbi decoding combines NNShot-derived token emission probabilities with the abstract transition distribution to select the best structured tag sequence.A temperature parameter re-normalizes transition probabilities so their scores align with the independently estimated emission scores.

4 Experiments

The experiments evaluate STRUCTSHOT and competing few-shot NER systems on tag set extension and domain transfer, using one-shot and five-shot settings across multiple domains. STRUCTSHOT consistently outperforms prior methods, with structured decoding, task-specific fine-tuning, and more support examples improving performance and reliability.

Experimental setup: The evaluation covers tag set extension and domain transfer across OntoNotes, CoNLL, I2B2, and WNUT, with one-shot and five-shot support sets.Tag set extension hides target entity tags during training, while domain transfer trains on OntoNotes and evaluates on News, Medical, and Social domains.
Results: STRUCTSHOT achieves new SOTA results, outperforming PrototypicalNet+P&D by 6% to 9% F1 in one-shot and 11% to 16% in five-shot settings.Across both few-shot NER tasks, NNShot and STRUCTSHOT perform significantly better than previous methods.
Results: Viterbi-based label-dependency modeling boosts NNShot by 2.4% F1 on five-shot tag set extension and 4% on five-shot domain transfer.These gains exceed those from joint CRF training with PrototypicalNet+P&D, supporting independent transition and emission scores as a cheap alternative.
Results: BERT-based systems outperform BiLSTM systems, while OntoNotes fine-tuning raises NNShot performance over SimBERT by 20% to 35% F1 across settings.The results indicate that task-specific information complements general language-model pretraining for few-shot NER.
Results: Five-shot systems are more accurate and robust than one-shot systems, with lower standard deviations across support sets.Domain transfer is generally harder than tag set extension because of language variation, although overlapping entity classes can benefit domain-transfer performance.
Experimental analysis: Using IO tagging consistently outperforms BIO, and standard NER-loss fine-tuning beats prototypical-network-loss fine-tuning for STRUCTSHOT.STRUCTSHOT and NNShot particularly benefit from switching from BIO to IO; standard NER-loss fine-tuning improves five-shot results by 12% and 9% on the reported tasks.

5 Discussion

STRUCTSHOT is effective because fine-tuning produces task-aware token representations while nearest-neighbor decoding handles the diffuse O class. Its main remaining weakness is distinguishing ambiguous entity types without domain-specific cues.

t-SNE visualization: Fine-tuning BERT on OntoNotes makes same-class instances much closer on CoNLL and WNUT than with pre-trained BERT, while class separation is stronger on CoNLL.The greater tag-set overlap between CoNLL and OntoNotes makes different entity classes more separable.
t-SNE visualization: O-labeled instances remain spread across the representation space, making a locally focused nearest-neighbor classifier suitable for assigning O.This representation behavior explains why STRUCTSHOT can handle the O class more effectively than prototype-based methods.
Per-class performance analysis: STRUCTSHOT performs reasonably on less ambiguous classes such as DATE, CITE, person, and location, but struggles with highly ambiguous entity types.It confuses numerical categories such as AGE, MEDICAL-RECORD, PHONE, and IDNUM, and often predicts PATIENT as DOCTOR or corporation for group.
Per-class performance analysis: Domain-specific cues such as “Dr.” and “MD.” may help resolve entity ambiguities and improve few-shot NER generalization.The proposed cues specifically address confusions that remain after transfer across domains.

6 Related Work

Prior few-shot NER work adapts metric-learning methods such as prototypical and matching networks, but these systems require complex episode training and achieve unsatisfactory transfer performance. STRUCTSHOT instead uses nearest-neighbor classification with structured decoding and avoids meta-training.

Meta learning: Meta-learning methods represent unseen classes with prototypes or learned metric spaces from few-shot support examples.Prototypical networks classify test points by distance to class prototypes, while nearest-neighbor feature transformations can also compete with meta-learning methods.
Few-shot NER: Prior few-shot NER systems apply prototypical or matching networks, with some also learning transition scores for label dependencies.These approaches include Fritzler et al. (2019) and Hou et al. (2020).
Few-shot NER: Existing few-shot NER approaches require complex episode training and achieve unsatisfactory results, whereas STRUCTSHOT needs no meta-training.STRUCTSHOT combines a simple nearest-neighbor classifier with a structured decoder.
Few-shot NER: STRUCTSHOT is reported to be more accurate than existing meta-learning-based few-shot NER systems.Its comparison is motivated by evaluating nearest-neighbor classification against established meta-learning methods.

7 Conclusion

The paper introduces STRUCTSHOT, a few-shot NER system that combines nearest-neighbor learning with structured decoding without few-shot-specific training. It reports state-of-the-art performance across multiple-domain benchmarks and proposes a standard evaluation setup.

7 Conclusion: STRUCTSHOT achieves state-of-the-art few-shot NER performance without any few-shot-specific training.The system addresses prior weaknesses in handling the O class and modeling label dependencies through nearest-neighbor learning and structured decoding.
7 Conclusion: The paper proposes a standard few-shot NER evaluation setup and identifies handling the O class and label dependencies as weaknesses of previous systems.These weaknesses motivate the use of nearest-neighbor learning and structured decoding.
7 Conclusion: STRUCTSHOT significantly outperforms prior state-of-the-art systems on popular few-shot NER benchmarks spanning multiple domains.The conclusion frames this result as evidence from the paper’s proposed evaluation setup.
7 Conclusion: Future work will extend the approach to other few-shot sequence-tagging problems such as part-of-speech tagging and slot filling.

A Appendix: Episode Evaluation Results

The appendix reports episode-evaluation results for one-shot and five-shot NER across tag-set extension and domain-transfer tasks. Across these evaluations, STRUCTSHOT maintains the same pattern of significantly outperforming competing systems.

Appendix: Episode Evaluation Results: STRUCTSHOT significantly outperforms all competitors in both one-shot and five-shot episode evaluations.The appendix states that this performance trend matches the main results.
Appendix: Episode Evaluation Results: The one-shot and five-shot episode evaluations use 100 evaluation episodes for each experiment.
Appendix: Episode Evaluation Results: The episode results cover both tag-set extension and domain-transfer tasks for one-shot and five-shot NER.Table 5 reports one-shot results, while Table 6 reports five-shot results.
Appendix: Episode Evaluation Results: Table 5 and Table 6 report F1 scores, with standard deviations computed from runs using five sampled support sets.The best results are marked in bold in the tables.