PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Named Entity Recognition as Word-Word Relation Classification

Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, Fei Li

arXiv:2112.10070v1cs.CL

TL;DR

Unified NER must handle flat, overlapped, and discontinuous entities in one system, while existing approaches have limitations in representing these structures and their word-level connectivity. W^2NER recasts the task as word-word relation classification using NNW and THW-* relations, a 2D word-pair grid, multi-granularity convolutions, and co-prediction. It achieves state-of-the-art performance across 14 benchmark datasets.

  • Problem

    Unified NER seeks to handle flat, overlapped, and discontinuous entities concurrently, but existing methods have limitations in modeling their neighboring relations and unified structure.

  • Method

    W^2NER formulates unified NER as word-word relation classification with NNW and THW-* relations, refined 2D word-pair grids, and a co-predictor.

  • Results

    W^2NER achieves state-of-the-art performance across 14 widely used benchmark datasets for flat, overlapped, and discontinuous NER.

  • Takeaways & Limitations

    The relation-classification formulation provides one framework for extracting flat, overlapped, and discontinuous entities simultaneously.

  • Takeaways & Limitations

    Discontinuous NER datasets include flat and overlapped entities as well, so those evaluations are not isolated to discontinuous entities.

Abstract

from arXiv · show

So far, named entity recognition (NER) has been involved with three major types, including flat, overlapped (aka. nested), and discontinuous NER, which have mostly been studied individually. Recently, a growing interest has been built for unified NER, tackling the above three jobs concurrently with one single model. Current best-performing methods mainly include span-based and sequence-to-sequence models, where unfortunately the former merely focus on boundary identification and the latter may suffer from exposure bias. In this work, we present a novel alternative by modeling the unified NER as word-word relation classification, namely W^2NER. The architecture resolves the kernel bottleneck of unified NER by effectively modeling the neighboring relations between entity words with Next-Neighboring-Word (NNW) and Tail-Head-Word-* (THW-*) relations. Based on the W^2NER scheme we develop a neural framework, in which the unified NER is modeled as a 2D grid of word pairs. We then propose multi-granularity 2D convolutions for better refining the grid representations. Finally, a co-predictor is used to sufficiently reason the word-word relations. We perform extensive experiments on 14 widely-used benchmark datasets for flat, overlapped, and discontinuous NER (8 English and 6 Chinese datasets), where our model beats all the current top-performing baselines, pushing the state-of-the-art performances of unified NER.

1 Introduction

Unified NER addresses flat, overlapped, and discontinuous entities together, but existing approaches face difficulties representing all subtasks and modeling neighboring relations between entity words. W^2NER reframes the problem as word-word relation classification and achieves state-of-the-art results across 14 benchmark datasets.

  • Motivation: Unified NER jointly handles flat, overlapped, and discontinuous entities, which were largely studied as separate subtasks.Overlapped entities can share tokens, while discontinuous entities contain non-adjacent spans.
  • Motivation: Existing sequence-labeling, hypergraph, sequence-to-sequence, and span-based methods face limitations including inflexible tagging schemes, exposure bias, span-length constraints, and high complexity.Span-based approaches enumerate candidate spans, which can be problematic for long-span entities.
  • Motivation: Neighboring relations between entity words are identified as the unified NER bottleneck because they connect partial segments, especially in overlapping and discontinuous entities.For example, recognizing “aching in shoulders” requires relating the segments “aching in” and “shoulders.”
  • Approach: W^2NER models both entity boundaries and neighboring relations through Next-Neighboring-Word and Tail-Head-Word-* relations.NNW captures adjacency within an entity, while THW-* links tail and head boundaries and encodes entity type.
  • Approach: The neural framework builds a word-pair grid, refines it with multi-granularity 2D convolutions, and uses a co-predictor to reason over word-word relations.BERT and BiLSTM provide contextualized word representations before grid construction.
  • Results: 14 benchmark datasets cover flat, overlapped, and discontinuous NER across English and Chinese, and W^2NER achieves the best performance against the evaluated baselines on all datasets.The comparison includes 12 flat-NER, 7 overlapped-NER, and 7 discontinuous-NER baselines.

2 NER as Word-Word Relation Classification

The paper formalizes unified NER as classification over relations between every pair of input words. NNW relations connect successive entity words, while THW-* relations identify entity boundaries and types, enabling simultaneous decoding of flat, overlapped, and discontinuous entities.

  • Formalization: Given N input words, the task extracts predefined relations between every token pair: NONE, NNW, or THW-*.The relations are represented over a word-pair grid.
  • Relation Types: NNW indicates that a row word has a successive word in the same entity mention.It captures neighboring links among entity words.
  • Relation Types: THW-* indicates that the row word is an entity tail, the column word is its head, and * specifies the entity type.This relation encodes boundary and category information.
  • Decoding: The relation design supports simultaneous extraction of flat, overlapped, and discontinuous entities.For example, NNW links connect “aching in” with either “legs” or “shoulders,” while THW relations connect each tail to “aching.”

3 Unified NER Framework

The framework represents unified NER as a 2D grid of directional word-pair relations, refines this grid with convolutions, and jointly predicts relations with MLP and biaffine components. Decoding NNW paths and THW relations yields flat, nested, overlapped, and discontinuous entity mentions.

  • Encoder Layer: BERT and BiLSTM encode contextualized word representations that initialize a directional N×N word-pair grid.Each grid cell represents an ordered pair of input words.
  • Convolution Layer: Conditional layer normalization generates each word-pair representation using one word as a condition and the other as the normalized input.The gain and bias parameters depend on the conditioning word representation.
  • Convolution Layer: Multi-granularity dilated 2D convolutions capture interactions between close and distant words in the word-pair grid.Different dilation rates produce multiple refined grid representations that are concatenated.
  • Convolution Layer: BERT-style grid construction adds word, relative-position, and relative-distance information before convolutional refinement.The resulting grid is enriched with pairwise positional information.
  • Co-Predictor Layer: The co-predictor combines an MLP over grid representations with a biaffine predictor over encoder representations to classify word-pair relations.The final relation probabilities combine the two predictors’ relation distributions.
  • Co-Predictor Layer: NNW relations define directional paths corresponding to entity mentions, while THW relations provide boundary, type, and disambiguation information across four decoding cases.Together, these relations support flat, nested, overlapped, and discontinuous entity structures.

4 Experimental Settings

The experiments cover flat, overlapped, and discontinuous NER across 14 benchmark datasets in English and Chinese, using established data splits and comparison settings.

  • Datasets: The evaluation spans 14 datasets: six for flat NER, five for overlapped NER, and three for discontinuous NER.The datasets include English and Chinese corpora, with discontinuous datasets drawn from biomedical or clinical documents.
  • Flat NER Datasets: Flat NER experiments use CoNLL-2003, OntoNotes 5.0, OntoNotes 4.0, MSRA, Weibo, and Resume.The first two are English datasets, while the latter four are Chinese datasets.
  • Overlapped NER Datasets: Overlapped NER experiments use ACE 2004, ACE 2005, and GENIA, with dataset-specific train/dev/test splits.GENIA uses five entity types and an 8.1:0.9:1.0 split; Chinese ACE datasets use an 8.0:1.0:1.0 split.
  • Discontinuous NER Datasets: Discontinuous NER experiments use CADEC, ShARe13, and ShARe14, where around 10% of entities are discontinuous.These datasets are derived from biomedical or clinical documents and use preprocessing scripts from Dai et al. (2020).

5 Experimental Results

W2NER achieves state-of-the-art results across flat, overlapped, and discontinuous NER benchmarks, while ablations support the contributions of its convolutional and relation-prediction components.

  • Flat NER Results: 93.07% F1 and 90.50% F1 are achieved on CoNLL-2003 and OntoNotes 5.0, respectively.On Chinese datasets, W2NER improves over previous state-of-the-art results by 0.27%, 0.01%, 0.54%, and 1.82% on OntoNotes 4.0, MSRA, Resume, and Weibo.
  • Overlapped NER Results: 87.52%, 86.79%, and 81.39% F1 are obtained on ACE2004, ACE2005, and GENIA, respectively.These results outperform tagging-based, span-based, hypergraph-based, and sequence-to-sequence approaches.
  • Discontinuous NER Results: 0.81%, 0.02%, and 0.45% F1 improvements over previous best models are reported on CADEC, ShARe13, and ShARe14, respectively.The authors describe these as new state-of-the-art results for discontinuous NER.
  • Mention-Type Analysis: W2NER achieves the best results for both overlapped and discontinuous mentions when those entities are isolated within the discontinuous-NER datasets.The comparison includes clique-based, sequence-to-sequence, and transition-based baselines.
  • Ablation Studies: Removing all convolutions causes obvious performance drops, supporting the usefulness of multi-granularity dilated convolution.Removing region and distance embeddings causes slight drops, while removing different dilation rates also decreases performance, especially dilation rate 2.
  • Ablation Studies: Removing NNW relations drops F1 on all datasets, including a 4.2% decrease on CADEC.The authors attribute this to discontinuous spans being incorrectly recognized as continuous without NNW relations.
  • Ablation Studies: The biaffine predictor contributes at most about 0.5% gains alongside the leading MLP predictor.This comparison evaluates the two predictors within the co-predictor layer.

6 Related Work on NER

Prior unified-NER methods mainly use tagging, hypergraphs, sequence-to-sequence generation, or span classification, each with representation or complexity constraints.

  • Sequence Labeling Approaches: Sequence-labeling approaches assign each token a tag using schemes such as BIO, BIOHD, or BIEOS.Designing one tagging scheme for all NER subtasks is difficult.
  • Sequence-to-Sequence Approaches: Sequence-to-sequence methods generate entity representations or spans as outputs.They form one of the major categories of prior multi-type NER methods.
  • Hypergraph-Based Approaches: Hypergraph-based approaches represent and infer entity mentions with hypergraphs.Lu and Roth (2015) introduced the hypergraph model for overlapped NER.
  • Span-Based Approaches: Span-based methods enumerate possible spans and classify them, but maximal span lengths and enumeration can increase model complexity.These limitations are especially relevant for long-span entities.

7 Conclusion

The paper concludes that W2NER unifies flat, overlapped, and discontinuous NER through word-word relation classification and achieves state-of-the-art performance across 14 datasets.

  • Conclusion: W2NER jointly models entity boundary identification and neighboring relations between entity words.Its relations are defined as next-neighboring-word and tail-head-word relations.
  • Conclusion: The framework uses a BERT-BiLSTM encoder, a convolution layer for the word-pair grid, and a co-predictor for jointly reasoning relations.The architecture also includes multi-granularity 2D convolutions and biaffine and MLP predictors.
  • Conclusion: W2NER achieves state-of-the-art performance on 14 widely used benchmark datasets.The conclusion also reports that the convolution-centric model and proposed modules are effective in ablation studies.
  • Conclusion: The decoding algorithm extracts entity index sequences by selecting THW-* relations and tracing NNW paths from head words to tail words.Single-token entities can be decoded directly from THW relations.

B Implementation Details

The experiments use domain-specific pretrained language models, a shared optimizer and PyTorch implementation, with hyperparameters tuned on the datasets.

  • BioBERT is used for GENIA and CADEC, Clinical BERT for ShARe13 and ShARe14, and vanilla BERT for the remaining datasets.
  • The model is optimized with AdamW.
  • The implementation uses PyTorch and training runs on an NVIDIA RTX 3090 GPU.

C Evaluation Metrics

Evaluation uses precision, recall and F1-score under exact token-sequence-and-type matching, with results averaged across five runs.

  • Precision, recall and F1-score are used as evaluation metrics.
  • A prediction is a true positive only when its token sequence and entity type exactly match the gold entity.
  • Each experiment is run five times, and the reported value is the average.
  • Parameter counts and training and inference speeds are compared for three baselines and the proposed model using PyTorch on an NVIDIA RTX 3090 GPU.

E Supplemental Experiments for Recognizing Overlapped or Discontinuous Entities

Supplemental experiments evaluate overlapped and discontinuous entity recognition on CADEC and ShARe13, where the proposed model ranks first and supports its relation-modeling motivation.

  • The proposed model ranks first on both CADEC and ShARe13.
  • Table 9’s notation uses “Ovlp.” for overlapped mentions and “Dis.” for discontinuous mentions.
  • Figure 6 reports overlapped and discontinuous mention results on CADEC and ShARe13.
  • These experiments support modeling both boundary-word relations and inside-word relations for recognizing overlapped or discontinuous entities.

F Dataset Statistics

The evaluation covers 14 datasets spanning English and Chinese flat, overlapped and discontinuous NER, with discontinuous datasets containing all three entity kinds.

  • The framework is evaluated on 8 English datasets and 6 Chinese datasets across three NER subtasks.
  • English flat NER uses CoNLL-2003 and OntoNotes 5.0, while Chinese flat NER uses OntoNotes 4.0, MSRA, Weibo and Resume.
  • English overlapped NER includes ACE 2004, ACE2005 and GENIA, while Chinese overlapped NER includes ACE 2004 and ACE 2005.
  • CADEC, ShARe13 and ShARe14 provide the English discontinuous NER datasets.
  • The three discontinuous NER datasets include all three kinds of entities.
Loading 2112.10070v1…