Source-linked AI summary
Named Entity Recognition as Dependency Parsing
Juntao Yu, Bernd Bohnet, Massimo Poesio
TL;DR
NER research often overlooks nested entities, where one entity contains another. The paper reformulates NER as span-based structured prediction with biaffine scoring and constraints, achieving state-of-the-art results across eight nested and flat corpora, with gains up to 2.2% absolute.
Problem
NER research often focuses on flat entities, despite named entities also occurring in nested structures.
Method
The system uses contextual embeddings, a BiLSTM, and a biaffine model to score all spans, then applies constraints for nested or flat NER.
Results
The system achieves state-of-the-art results on all eight evaluated corpora, with improvements of up to 2.2% absolute percentage points.
Takeaways & Limitations
Advanced structured prediction techniques substantially improve both nested and flat NER.
Abstract
from arXiv · showhide
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, concerned with identifying spans of text expressing references to entities. NER research is often focused on flat entities only (flat NER), ignoring the fact that entity references can be nested, as in [Bank of [China]] (Finkel and Manning, 2009). In this paper, we use ideas from graph-based dependency parsing to provide our model a global view on the input via a biaffine model (Dozat and Manning, 2017). The biaffine model scores pairs of start and end tokens in a sentence which we use to explore all spans, so that the model is able to predict named entities accurately. We show that the model works well for both nested and flat NER through evaluation on 8 corpora and achieving SoTA performance on all of them, with accuracy gains of up to 2.2 percentage points.
1 Introduction
NER commonly focuses on flat entities, although named entities can be nested. The paper introduces one biaffine dependency-parsing-inspired system for both nested and flat NER.
- Nested entities contain references to other named entities, as in [Bank of [China]], where both spans are entities.Nested entities occur frequently in ACE 2004, ACE 2005, and GENIA, while CONLL 2002, CONLL 2003, and ONTONOTES contain only flat entities.
- The paper reformulates NER as identifying span start and end indices and assigning each span a category.It adopts ideas from the biaffine dependency parsing model to score candidate spans.
- The biaffine model scores all possible spans, after which candidate spans are ranked and selected under flat or nested NER constraints.This gives the system a shared approach to both entity structures.
- The system achieves state-of-the-art results on three nested NER corpora and five flat NER corpora, with gains of up to 2.2% absolute percentage points.
2 Related Work
Prior NER work largely uses sequence labeling for flat entities, while nested NER has been addressed with iterative, parsing-based, and span-based neural models. The paper distinguishes its approach through biaffine scoring of spans and a global sentence view.
- Flat Named Entity Recognition: Most flat NER systems use neural sequence labeling, commonly combining CNN or LSTM encoders with CRF prediction.Later systems incorporate contextual embeddings such as ELMo and BERT.
- Flat Named Entity Recognition: Lample et al. cast flat NER as transition-based dependency parsing with Stack-LSTM and shift/reduce transitions.Their transition-based system handles flat NER, whereas the paper’s system targets both flat and nested entities.
- Nested Named Entity Recognition: Nested NER research includes constituency parsing, iterative LSTM-CRF prediction, and exhaustive span enumeration.
- Nested Named Entity Recognition: The paper uses a biaffine model to obtain a global sentence view, unlike span representations formed from start and end outputs up to a defined length.
3 Methods
The model encodes tokens with word and character representations, derives separate start and end representations, and uses biaffine scoring to classify every valid span. Ranking and structural constraints then produce nested or flat entities.
- Word and character embeddings are processed by a BiLSTM, followed by a biaffine classifier inspired by dependency parsing.The word inputs include BERTLarge and fastText embeddings, while a CNN encodes token characters.
- Separate FFNNs produce start and end representations so the model can learn their distinct contextual roles.The paper reports that this improves accuracy over directly using LSTM outputs.
- The biaffine model creates an l×l×c scoring tensor over sentence positions and NER categories plus non-entity.Here l is sentence length and c is the number of categories plus one.
- The score tensor covers all spans with start index before or equal to end index, and each span receives a NER category.The model therefore treats valid-span categorization as a multi-class classification problem.
- Candidate spans are ranked by category scores, then filtered by boundary constraints for nested or flat NER.Nested NER rejects boundary clashes with higher-ranked entities, while flat NER additionally rejects containment.
4 Experiments
The experiments evaluate the system on nested and flat NER corpora using standard precision, recall, and F1, with dataset-specific splits and shared experimental settings.
- Data Set: The evaluation covers ACE 2004, ACE 2005, and GENIA for nested NER, plus CONLL 2002, CONLL 2003, and ONTONOTES for flat NER.CONLL evaluations span English, German, Dutch, and Spanish, while ONTONOTES uses English.
- Data Set: ACE 2004 and ACE 2005 use 80%/10%/10% train, development, and test splits based on prior settings and documents.GENIA uses a 90%/10% train/test split without a development set.
- Evaluation Metric: The experiments report recall, precision, and F1, counting an entity as correct only when both its boundary and category are correct.
- Hyperparameters: A unified setting is used across experiments, with hyperparameters reported in Table 1.
- Results: Table 2 compares state-of-the-art performance on ACE 2004, ACE 2005, and GENIA for nested NER.
5 Results on Nested NER
The system achieves state-of-the-art results on nested NER across ACE 2004, ACE 2005, and GENIA, with improvements over prior systems. Its strongest reported gain is on GENIA.
- 2% and 1.1% improvements over previous state of the art were achieved on ACE 2004 and ACE 2005, respectively.
- 80.5% F1 was achieved on GENIA, improving the state of the art by 2.2% absolute.
- The system achieved state-of-the-art results on all three nested NER corpora.
6 Results on Flat NER
The system performs strongly on flat NER across multilingual and domain-diverse corpora, remaining comparatively stable across domain and category-granularity differences.
- 86.4% F1 was achieved for German, 90.3% for Spanish, and 93.5% for Dutch on multilingual flat NER.
- The system exceeded previous state-of-the-art results by 2.1% on ONTONOTES, 1.5% on Spanish, 1.3% on German, and 1% on Dutch.It was slightly better than the state of the art on English and gained 2% against Akbik et al. (2018) on revised German data.
7 Ablation Study
Ablations on ONTONOTES show that biaffine mapping and BERT embeddings contribute most to the model's accuracy, while fastText contributes modestly and character embeddings have little impact.
- Replacing the biaffine mapping with a CRF layer reduced performance by 0.8 percentage points.This converts the system into a sequence labelling model.
- Removing BERT embeddings reduced performance by 2.4 percentage points, making them one of the most important accuracy factors.
- Removing context-independent fastText embeddings reduced the score by 0.4%.The authors report that this contribution remains noticeable even with BERT enabled.
- Character embeddings had a quite small impact on performance.The authors relate this to English morphology and BERT's word-piece representations.
- Biaffine mapping and BERT embeddings together contributed most to the system's high accuracy.
8 Conclusion
The paper reformulates NER as structured prediction using dependency-parsing techniques for both nested and flat entities. Evaluated on eight corpora, the system achieves state-of-the-art performance across all of them.
- The system uses contextual embeddings, a multi-layer BiLSTM, and a biaffine model to score all spans in a sentence.Constraints guide predictions for nested or flat named entities.
- The model achieves state-of-the-art performance on all eight evaluated named entity corpora.
- Advanced structured prediction techniques substantially improve performance for both nested and flat NER.