Source-linked AI summary
Few-NERD: A Few-Shot Named Entity Recognition Dataset
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu
TL;DR
Few-shot NER lacks dedicated benchmarks and often targets coarse-grained types despite the prevalence of fine-grained unseen entities. FEW-NERD introduces a large human-annotated fine-grained dataset with unified generalization benchmarks, and experiments show that few-shot NER remains challenging and warrants further research.
Problem
Few-shot NER lacks dedicated benchmark data, while existing datasets emphasize coarse-grained types although unseen entities are often fine-grained.
Method
The paper constructs FEW-NERD, a human-annotated dataset with 8 coarse-grained and 66 fine-grained types, and designs three unified benchmark tasks.
Results
Few-NERD is challenging across supervised, intra-type, and inter-type settings, while providing correlation information for few-shot learning.
Takeaways & Limitations
FEW-NERD provides a unified resource intended to facilitate further research on few-shot NER.
Takeaways & Limitations
FEW-NERD is annotated in English, limiting its linguistic coverage.
Abstract
from arXiv · showhide
Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practical and challenging task. Current approaches collect existing supervised NER datasets and re-organize them to the few-shot setting for empirical study. These strategies conventionally aim to recognize coarse-grained entity types with few examples, while in practice, most unseen entity types are fine-grained. In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238 sentences from Wikipedia, 4,601,160 words are included and each is annotated as context or a part of a two-level entity type. To the best of our knowledge, this is the first few-shot NER dataset and the largest human-crafted NER dataset. We construct benchmark tasks with different emphases to comprehensively assess the generalization capability of models. Extensive empirical results and analysis show that Few-NERD is challenging and the problem requires further research. We make Few-NERD public at https://ningding97.github.io/fewnerd/.
1 Introduction
Few-shot NER lacks dedicated, consistent benchmarks despite the practical need to recognize difficult unseen entity types. FEW-NERD addresses this gap with a large, fine-grained dataset and unified evaluation tasks.
- Few-shot NER is important because professionally specialized entities are difficult to annotate at large scale.
- Existing few-shot NER studies reorganize supervised datasets that typically provide only 4–18 coarse-grained entity types.
- Most unseen entities are fine-grained, exposing a mismatch between conventional datasets and practical few-shot NER.
- 188.2k sentences and 491.7k manually annotated entities form FEW-NERD, organized into 8 coarse-grained and 66 fine-grained types.
- The dataset supplies contextual and cross-type correlation information, while empirical results show that all three settings remain challenging.
- FEW-NERD provides standard supervised, intra-type, and inter-type tasks for assessing instance-level, type-level, and transfer generalization.
2 Related Work
Prior work established broad NER resources and increasing interest in few-shot NER, but dedicated benchmarks and fine-grained evaluation remain limited.
- CoNLL’03 is a widely used Reuters News dataset containing 4 coarse-grained entity types.
- NER research has expanded across datasets from multiple domains, alongside strong deep-neural performance under sufficient supervision.
- Few-shot NER has gained attention because it addresses challenging recognition with few examples and limited supervision.
- Existing studies use inconsistent few-shot settings because no specific benchmark dataset was available.
- FEW-NERD responds by combining a fine-grained annotation schema with benchmarks designed to assess model generalization.
3 Problem Formulation
Few-shot NER formulates entity recognition as sequence labeling under support-query supervision, but sentence-level sampling and dense entities complicate episode construction.
- 3.1 Named Entity Recognition: NER assigns each token in an input sequence a predefined entity-type label or the non-entity O label.
- 3.2 Few-shot Named Entity Recognition: N-way K-shot episodes sample N classes and K examples per class to form a support set.
- 3.2 Few-shot Named Entity Recognition: A query set supplies K′ examples for each of the N sampled classes.
- 3.2 Few-shot Named Entity Recognition: Training predicts query labels using support information, while testing uses labeled support examples for classes unseen during training.
- 3.2 Few-shot Named Entity Recognition: NER episodes must sample complete sentences because one sentence can contain entities from multiple classes and context is crucial.
- 3.2 Few-shot Named Entity Recognition: Greedy sampling becomes increasingly restrictive as support classes approach their shot limits, especially for dense-entity data.
- 3.2 Few-shot Named Entity Recognition: FEW-NERD uses an N-way K∼2K-shot setting so each class receives between K and 2K examples, alleviating sampling constraints.
- 3.2 Few-shot Named Entity Recognition: The algorithm takes dataset X, label set Y, N, and K as inputs and returns an output result.
4 repeat
The passage marks a computation involving entity-type counts after an earlier algorithmic step.
- The procedure computes the size of Count and Counti after a preceding step.
- Count and Counti are the quantities referenced by the algorithm at this stage.
- This step occurs after another algorithmic operation whose details are not stated in the passage.
4 Collection of FEW-NERD
FEW-NERD is built from Wikipedia with a two-level entity schema designed for fine-grained few-shot NER. Paragraph selection, contextual annotation, and expert review support balanced, consistent labels.
- 4.1 Schema of Entity Types: The finalized schema contains 8 coarse-grained and 66 fine-grained entity types.The schema is adapted from FIGER and refined through frequency analysis, pre-annotation, and manual denoising.
- 4.2 Paragraph Selection: Wikipedia supplies a large corpus with varied entities and rich contextual information.The raw corpus is the entire English Wikipedia dump.
- 4.2 Paragraph Selection: 66,000 paragraphs are selected across all 66 fine-grained types, averaging 61.3 tokens per paragraph.Selection uses manually denoised dictionaries built from entity mentions annotated in FIGER, with category balance as a goal.
- 4.3 Human Annotation: Context determines labels in difficult cases, such as assigning “London” to Art-Music rather than Location-GPE.The example illustrates why annotation requires linguistic training and contextual judgment.
- 4.3 Human Annotation: Each paragraph is independently annotated by two trained annotators and reviewed by an experienced expert.The annotation team includes 70 annotators and 10 experienced experts.
- 4.3 Human Annotation: 76.44% indicates a high degree of annotation consistency.The supplied passage reports this result as the outcome of the consistency assessment.
5 Data Analysis
FEW-NERD is larger and more diverse than several widely used NER datasets, while its type-similarity analysis examines the knowledge-transfer structure available for few-shot learning.
- 5.1 Dataset Statistics: FEW-NERD is significantly larger than the compared NER datasets and contains more entity types and annotated entities.The comparison includes CoNLL’03, WikiGold, OntoNotes 5.0, WNUT’17, and I2B2.
- 5.1 Dataset Statistics: The dataset is balanced because paragraphs are selected to avoid severely uneven fine-grained type distributions.Location, especially GPE, and Person are among the most frequent entity types.
- 5.2 Knowledge Correlations among Types: The knowledge-correlation analysis trains a BERT-Tagger, selects its model, and represents mentions from the remaining 20% of FEW-NERD data.The analysis uses 70% of the data for training and 10% for model selection.
- 5.2 Knowledge Correlations among Types: Each heat-map square represents the similarity between two entity types.The figure visualizes knowledge correlations among FEW-NERD types.
- 5.2 Knowledge Correlations among Types: Most fine-grained types across coarse-grained categories share little correlation because their contextual features differ.The result motivates a benchmark setting focused on knowledge transfer.
6 Benchmark Settings
FEW-NERD defines supervised and few-shot benchmark tasks that test different forms of NER generalization. The few-shot settings partition entity types and mask entities from other partitions to prevent exposure to new types during training.
- Benchmark Overview: 188,238 sentences with 66 fine-grained types support three FEW-NERD benchmark tasks.The benchmarks are constructed to exploit entity and contextual information while evaluating generalization from different perspectives.
- 6.1 Standard Supervised NER: FEW-NERD (SUP) randomly splits data into 70% training, 10% validation, and 20% testing, with all 66 types present in each split.This setting assesses instance-level generalization and remains challenging because of the large number of entity types.
- 6.2 Few-shot NER: Few-shot benchmarks split the entity set into mutually disjoint training, development, and test subsets.Instances in each dataset contain entities from its assigned subset, while entities from other subsets are replaced with O.
- 6.2 Few-shot NER: FEW-NERD (INTRA) assigns entire coarse-grained categories to different partitions to test prediction of new entities across coarse types.People, MISC, Art, and Product are assigned to training; Event and Building to development; ORG and LOC to testing.
- 6.2 Few-shot NER: Table 3 reports train, development, and test statistics for the three tasks after removing sentences without entities from few-shot benchmarks.The table summarizes the benchmark splits rather than the full sentence collection.
7 Experiments
The experiments evaluate BERT-based supervised and few-shot NER models across FEW-NERD’s benchmark settings, showing that fine-grained entity recognition remains challenging. Performance varies by entity type, sampling difficulty, task split, and modeling approach.
- 7.1 Models: BERT-Tagger, ProtoBERT, NNShot, and StructShot provide representative BERT-based baselines for supervised and few-shot NER.BERT-Tagger uses a linear classifier, ProtoBERT computes entity-type prototypes, and NNShot and StructShot use token-level similarity.
- 7.1 Models: Precision, recall, and micro F1-score are calculated over test episodes using the IO labeling schema.The IO schema labels all entity tokens as I-type and other tokens as O.
- 7.2 The Overall Results: 91.34% and 89.11% are the reported F1-scores on two widely used supervised NER datasets, while BERT-Tagger suffers a grave drop on FEW-NERD (SUP).The authors attribute the FEW-NERD difficulty partly to its larger number of entity types.
- 7.2 The Overall Results: Person achieves the best performance and Product the worst, while Coarse-Other has the lowest F1-score for almost all coarse-grained types.Performance is not linearly correlated with type frequency; for example, Art performs well despite representing a small fraction of FEW-NERD.
- 7.2 The Overall Results: Models generally perform better on FEW-NERD (INTER) than FEW-NERD (INTRA), while 10 way 1∼2 shot is generally worst and 5 way 5∼10 shot best.INTRA is harder because training and test entity types share less knowledge across coarse-grained splits.
- 7.2 The Overall Results: ProtoBERT generally outperforms NNShot and StructShot, while StructShot’s Viterbi decoder improves precision in FEW-NERD (INTRA).The authors suggest prototype calculation provides regularization, whereas nearest-neighbor training may be unstable.
8 Conclusion and Future Work
The paper introduces FEW-NERD as a large-scale, fine-grained few-shot NER dataset and provides unified benchmarks showing that few-shot NER remains challenging.
- FEW-NERD is a large-scale few-shot NER dataset with fine-grained entity types.
- FEW-NERD is presented as the first few-shot NER dataset and one of the largest human-annotated NER datasets.
- Three unified benchmarks assess few-shot NER approaches across different evaluation settings.
- Empirical experiments demonstrate that few-shot NER remains challenging and merits further research.
Ethical Considerations
FEW-NERD is an English Wikipedia-based, human-annotated dataset whose construction combines automatic resources, manual denoising, and an annotation interface.
- FEW-NERD is a human-annotated few-shot NER dataset built from publicly obtained English Wikipedia content.
- The dataset is annotated in English, so its primary scope is English NLP research.
- The construction pipeline extracts and tokenizes Wikipedia text before indexing paragraphs for entity-rich selection.
- Fine-grained type dictionaries derived from FIGER were manually denoised before being used in data construction.
- Annotators selected entity spans and assigned corresponding coarse- and fine-grained types through an annotation interface.
B Implementation Details
The implementation uses BERT-based models and organizes FEW-NERD’s hierarchical schema with labeled examples spanning many entity categories.
- All four models use BERTbase with 768 hidden units, 12 layers, and 12 attention heads.
- Models are implemented with PyTorch and Hugging Face Transformers and optimized with AdamW at a learning rate of 1e-4.
- FEW-NERD’s schema contains coarse- and fine-grained entity types, including abbreviations such as ORG and MISC.
- The implementation materials include a table listing all FEW-NERD entity types and example annotations.
- Examples cover diverse categories, including locations, people, organizations, facilities, media, science, and events.