Source-linked AI summary
A Unified Generative Framework for Various NER Subtasks
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, Xipeng Qiu
TL;DR
Existing NER formulations struggle to handle flat, nested, and discontinuous entities concurrently without specialized schemas or span enumeration. The paper unifies these subtasks as entity span sequence generation with a BART-based Seq2Seq pointer framework and three entity representations. It reports SoTA or near SoTA performance on eight English NER datasets, while evaluations use existing public scientific-paper datasets and one comparison omits document context.
Problem
Existing token-level and span-level methods can hardly handle flat, nested, and discontinuous NER concurrently, requiring specialized tagging schemas or span enumeration.
Method
The paper formulates NER as entity span sequence generation using a unified Seq2Seq pointer framework with BART and three pointer-based entity representations.
Results
SoTA or near SoTA performance is reported across eight NER datasets covering flat, nested, and discontinuous subtasks.
Takeaways & Limitations
The unified formulation supports all three NER subtasks without specialized tagging-schema design or span enumeration.
Takeaways & Limitations
The experiments use existing datasets derived from public scientific papers, and one comparison is rerun with sentence context only rather than document context.
Abstract
from arXiv · showhide
Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-the-art (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.
1 Introduction
NER includes flat, nested, and discontinuous subtasks, but existing token- and span-based approaches struggle to address them uniformly. The paper proposes a unified Seq2Seq generation framework that represents entities as pointer sequences.
- NER comprises flat, nested, and discontinuous subtasks, with nested entities overlapping and discontinuous entities containing nonadjacent spans.
- Sequence labelling requires different tagging schemas across NER subtasks, while span-based methods enumerate quadratic numbers of spans and are especially difficult to apply to discontinuous NER.
- The proposed Seq2Seq framework directly generates entity pointer index sequences, allowing flat, continuous, and discontinuous entities to share one formulation.
- The framework incorporates BART and uses three entity representations to linearize entities without specialized tagging-schema design or span enumeration.
- The authors report SoTA or near SoTA performance across eight datasets spanning flat, nested, and discontinuous NER.
2 Background
NER research has developed token-level, span-level, and combined formulations, alongside Seq2Seq and pre-trained-model approaches. These lines of work differ in how they represent tags, spans, and contextual information.
- Token-level classification: Token-level NER assigns each token a tag derived from entity labels and schemes such as BIO or BILOU.
- Span-level classification: For nested and discontinuous NER, token-level tagging can become complex or multi-level, motivating direct span-level classification.
- Span-level classification: Span-level methods differ primarily in how they obtain candidate entity spans.
- Combined token-level and span-level classification: Combined approaches integrate token-level and span-level classification to avoid enumerating every possible span and incorporate entity-boundary information.
- Seq2Seq and pre-trained models: Seq2Seq models have been applied to NER by predicting entity starts, span lengths, and labels, while pre-trained models have motivated further approaches.
- BART: BART uses transformer encoder and decoder layers to recover original text from corrupted input and is available in base and large versions.
3 Proposed Method
The method uses a pointer-based Seq2Seq framework to generate entity index sequences, enabling unified decoding across NER subtasks. BART and three entity representations support unambiguous span linearization despite BPE tokenization.
- NER Task Formulation: Each entity is represented by span start and end indexes for one or more spans, followed by an entity tag index.The formulation supports single-span flat or nested entities and multi-span discontinuous entities.
- Seq2Seq Framework: The model encodes the input sentence and autoregressively generates an entity pointer-index sequence with a pointer-based decoder.The decoder predicts index distributions step by step and converts generated indexes into entity spans during inference.
- Seq2Seq Framework: The encoder produces sentence vectors, while the decoder uses Index2Token conversion because target sequences contain pointer and tag indexes.The converted tokens are used to obtain decoder hidden states and index probability distributions.
- BART Integration: BART is fine-tuned as the pretrained Seq2Seq backbone, with its encoder processing the input and its decoder generating the entity sequence.The method adopts BART because the task is formulated as Seq2Seq generation; no additional embeddings are used in the reported setup.
- Entity Representations: Three pointer-based representations locate entities under BPE tokenization: Span uses boundary BPEs, BPE lists all entity BPEs, and Word uses each entity word’s first BPE.Each representation appends the entity tag, and entities absent from the sentence are omitted from the target sequence.
4 Experiment
Experiments cover eight datasets spanning flat, nested, and discontinuous NER, using fine-tuned BART-Large and span-level F1 evaluation. The method is compared across these settings, with stronger performance reported on flat and discontinuous NER datasets.
- Datasets: Experiments evaluate eight datasets across flat, nested, and discontinuous NER subtasks.The datasets include CoNLL-2003 and OntoNotes; ACE 2004, ACE 2005, and Genia; and CADEC, ShARe13, and ShARe14.
- Evaluation protocol: Performance comparisons include a reproduction using only sentence-level context, reflecting the absence of document context in the reproduced setting.The table notation distinguishes rerun results and sentence-level-context reproductions.
- Datasets: Only Adverse Drug Events entities are considered in the discontinuous datasets because they contain the discontinuous annotations.The discontinuous datasets are CADEC, ShARe13, and ShARe14.
- Experimental settings: BART-Large is used for all experiments, with 12-layer encoder and decoder, no additional embeddings, and end-to-end fine-tuning.Performance is reported using span-level F1.
5 Results
The unified generative framework performs competitively across flat, nested, and discontinuous NER, while entity representation length and similarity to BART’s pre-training tasks influence performance.
- Our method achieves better performance than token-level and span-level baselines on both flat NER datasets.
- Our BART-based generative models are comparable to token-level and span-level classification models on three nested NER datasets.
- Our model achieves better performance than other models on all three discontinuous NER datasets.
- Comparison Between Different Entity Representations: The “Word” representation generally performs best because shorter entity representations are advantageous for generative frameworks.
- Comparison Between Different Entity Representations: Task similarity with continuous BPE prediction can outweigh representation length, explaining why “BPE” outperforms “Span” on several datasets.
- Comparison Between Different Entity Representations: “Span” is the clear outlier on Genia, where longer BPE tokenizations make the “Word” representation less similar to BART’s pre-training tasks.
6 Analysis
Analysis shows that the framework can recognize discontinuous entities despite their rarity, usually produces few invalid predictions, and exhibits position-dependent recall patterns.
- Recall of Discontinuous Entities: The model predicts discontinuous named entities and achieves better performance on the discontinuous entities themselves.
- Invalid Predictions: Invalid predictions are less than 1% in most cases for the “Word” representation, and they are excluded during evaluation.
- Entity Position: For flat and discontinuous NER, later entities have a larger probability of being recalled than earlier entities.
- Entity Position: Nested-NER recall is more involved because later outer entities may depend on correctly predicting earlier contained entities.
7 Conclusion
The paper presents a unified BART-based Seq2Seq formulation for diverse NER subtasks and reports SoTA or near-SoTA performance across eight datasets.
- NER subtasks are formulated as entity span sequence generation, enabling one pointer-based Seq2Seq model for flat, nested, and discontinuous NER.
- The framework tests three entity representations to linearize entity spans, with shorter and more continuous-BPE-like representations performing better.
- The proposed method achieves SoTA or near SoTA performance on eight NER datasets, demonstrating generality across NER subtasks.
Ethical Considerations
The ethical considerations describe the data sources, dataset analysis, identity-related scope, and computational-resource requirements of the experiments.
- All experiments use existing datasets derived from public scientific papers.
- The paper describes dataset characteristics and states that its analysis is consistent with the results.
- The work states that it contains no identity characteristics and does not harm anyone.
- The experiments are reported to require relatively few computer resources compared with pre-trained models.
A.1 Hyper-parameters
The experiments use dataset-specific hyper-parameters and slanted triangular learning-rate warmup, with all runs conducted on an Nvidia GeForce RTX-3090 GPU.
- A.1 Hyper-parameters: All experiments use slanted triangular learning-rate warmup and an Nvidia GeForce RTX-3090 with 24G graphical memory.
- A.1 Hyper-parameters: Dataset-specific hyper-parameters are listed for CoNLL2003, OntoNotes, ACE2004, ACE2005, Genia, CADEC, ShARe13, and ShARe14.The hyper-parameter settings are presented in Table 7.
A.2 Beam Search
Beam search has almost no effect on model performance, while the generative model trains faster than BERT-CRF but evaluates more slowly because inference is autoregressive.
- A.2 Beam Search: Beam search has almost no effect on model performance, and the F1 score changes little as beam size increases.The authors attribute the limited F1 effect possibly to the small search space during generation.
- A.2 Beam Search: The efficiency comparison measures memory footprint, training time, and evaluation time against BERT-MLP and BERT-CRF on CoNLL-2003 and OntoNotes.The comparison uses BERT-base and BART-base models, with batch sizes of 16 for training and 48 for evaluation.
- A.2 Beam Search: BERT-MLP predicts each token's tag using the largest probability, whereas BERT-CRF uses Viterbi decoding to find the highest-scoring label sequence.BERT-MLP uses a tag probability distribution, while BERT-CRF models valid label sequences and tag-transition scores.
- A.2 Beam Search: Causal masking enables parallel training, allowing the proposed model to train faster than BERT-CRF's sequential computation.
- A.2 Beam Search: Autoregressive generation makes evaluation slow, motivating future work on non-autoregressive decoding.