Source-linked AI summary
Neural Architectures for Nested NER through Linearization
Jana Straková, Milan Straka, Jan Hajič
TL;DR
Nested NER requires handling overlapping entities and multiple labels, a challenge addressed here by two neural architectures using linearized label encodings. The models surpass nested NER state of the art on four corpora, while contextual embeddings further improve nested and several flat NER results.
Problem
Nested NER must represent entities that overlap and may have multiple labels.
Method
The paper linearizes nested labels and models them with either LSTM-CRF multilabel prediction or token-conditioned seq2seq decoding.
Results
The proposed architectures surpass nested NER state of the art on ACE-2004, ACE-2005, GENIA and Czech CNEC, with contextual embeddings producing further improvements.
Takeaways & Limitations
The LSTM-CRF is better suited to less-nested or flat corpora, whereas seq2seq captures more complex nested relationships.
Takeaways & Limitations
The approach depends on a non-ambiguous enhanced BILOU encoding and uses no handcrafted classification features.
Abstract
from arXiv · showhide
We propose two neural network architectures for nested named entity recognition (NER), a setting in which named entities may overlap and also be labeled with more than one label. We encode the nested labels using a linearized scheme. In our first proposed approach, the nested labels are modeled as multilabels corresponding to the Cartesian product of the nested labels in a standard LSTM-CRF architecture. In the second one, the nested NER is viewed as a sequence-to-sequence problem, in which the input sequence consists of the tokens and output sequence of the labels, using hard attention on the word whose label is being predicted. The proposed methods outperform the nested NER state of the art on four corpora: ACE-2004, ACE-2005, GENIA and Czech CNEC. We also enrich our architectures with the recently published contextual embeddings: ELMo, BERT and Flair, reaching further improvements for the four nested entity corpora. In addition, we report flat NER state-of-the-art results for CoNLL-2002 Dutch and Spanish and for CoNLL-2003 English.
1 Introduction
Nested NER must handle overlapping entities and multiple labels, motivating two neural architectures that encode nested labels linearly. The models outperform nested NER state of the art across four corpora, with further gains from contextual embeddings.
- Nested NER handles entities that overlap and may receive more than one label.For example, “The Florida Supreme Court” contains the overlapping entity “Florida”.
- The paper proposes two neural architectures that infer nested-entity relationships without explicitly modeling a structural representation.
- The first architecture concatenates nested labels into multilabels predicted by a standard LSTM-CRF.This reuses an existing NER pipeline but increases the number of NE classes.
- The second architecture treats nested NER as sequence-to-sequence prediction, producing labels for each token until an end-of-word label is emitted.
- The models’ expressiveness depends on a non-ambiguous enhanced BILOU encoding of nested entity structure.
- The proposed methods surpass nested NER state of the art on ACE-2004, ACE-2005, GENIA and Czech CNEC.Adding ELMo, BERT and Flair yields further improvements on these nested corpora and state-of-the-art flat NER results on the listed CoNLL datasets.
2 Related Work
Prior nested NER methods explicitly represent nested structure with stacked recognizers, constituency graphs or hypergraphs. This paper instead encodes the structure as label sequences for implicit neural modeling.
- Finkel and Manning model nested structure as a syntactic constituency tree.
- Ju et al. run stacked LSTM-CRF recognition from innermost to outermost entities while at least one nested entity remains predicted.
- Wang and Lu build a hypergraph capturing all possible entity mentions in a sentence.
- Katiyar and Cardie use RNNs to model edge probabilities in a directed hypergraph.
- The proposed architectures avoid explicit structure construction and rely on neural networks to model relationships implicitly through label sequences.A related sequence-to-sequence parser predicts constituent hierarchy for lookahead features, but serves a different parsing purpose.
3 Datasets
The study evaluates nested NER on four corpora and flat NER on CoNLL datasets spanning English, German, Dutch, Spanish and Czech. Final models use train-plus-development data and strict span-and-class matching.
- Nested NER is evaluated on ACE-2004, ACE-2005, GENIA and Czech CNEC.
- The experiments follow prior train/dev/test splits, including a 90%/10% train/test split for GENIA.
- Flat NER is evaluated on CoNLL-2003 English and German and CoNLL-2002 Dutch and Spanish.
- Models are tuned on development data, retrained on concatenated train-plus-development data, and evaluated on the test set.
- Evaluation is strict: an entity is correct only when both its span and class are correct.
4 Methods
The method linearizes nested entities into per-token multilabel BILOU sequences and uses either LSTM-CRF or seq2seq decoding. The architectures combine word and character representations with optional contextual embeddings and no handcrafted classification features.
- 4.1 Nested NE BILOU Encoding: Nested entities are encoded in a CoNLL-like per-token BILOU representation.BILOU uses beginning, inside, unit-length, last and outside labels.
- 4.1 Nested NE BILOU Encoding: Overlapping mentions are concatenated into each token’s multilabel, prioritizing earlier starts and then longer mentions with the same start.
- 4.1 Nested NE BILOU Encoding: The encoding can be viewed as an unfolded directed hypergraph whose shared labels remain distinct and whose O label marks tokens outside mentions.
- 4.2 Neural Models for Nested NER: The LSTM-CRF model uses a bidirectional LSTM encoder and CRF decoder to predict nested multilabels.
- 4.2 Neural Models for Nested NER: The seq2seq model predicts labels one by one for each token, uses hard attention on that token, and stops with the <eow> label.
- 4.2 Neural Models for Nested NER: Input representations include word, lemma, POS and character-level embeddings, with optional ELMo, BERT or Flair contextual embeddings.BERT subword embeddings are averaged across its last four layers without finetuning.
- 4.2 Neural Models for Nested NER: The models use no handcrafted classification features.
5 Results
The seq2seq model clearly outperforms known nested NER methods, while contextual embeddings further improve results across nested and flat NER benchmarks.
- 86.88 F1 is reported for seq2seq+ELMo+BERT+Flair in the nested NER results.
- Table 1 reports nested NER F1 for ACE-2004, ACE-2005, GENIA, and Czech CNEC, with annotations identifying best and above-state-of-the-art results.The table notes a different ACE-2005 data split and distinguishes non-neural models.
- For shorter and less overlapping entities, the simplicity of LSTM-CRF wins over seq2seq, as in GENIA and flat corpora.
- Contextual embeddings produce substantial F1 increases across languages and corpora, though CoNLL-2003 German remains behind Akbik et al. (2018).The embeddings evaluated are ELMo, BERT, and Flair.
6 Conclusions
The paper presents two neural architectures and an enhanced BILOU encoding for nested entities. Seq2seq handles more complex nested relationships, while contextual embeddings improve results on nested and flat corpora.
- The paper presents two neural architectures and a simple encoding algorithm for modeling multiple named-entity labels in an enhanced BILOU scheme.
- LSTM-CRF multilabel modeling is better suited to less-nested and flat corpora, whereas seq2seq captures more complex nested relationships.
- Seq2seq surpasses the nested NER state of the art on four nested entity corpora.
- Contextual word embeddings yield state-of-the-art results on both nested and flat named-entity corpora.