SpanNER: Named Entity Re-/Recognition as Span Prediction

1 Introduction

• Span prediction and sequence labeling have complementary strengths: sequence labeling is better for long entities with low label consistency, while span prediction is better for sentences with more OOV words and medium-length entities.The paper frames this fine-grained comparison as an investigation of the paradigms’ strengths and weaknesses.Source: §1 · p. 2
• A span prediction model can serve both as a base NER system and as a meta-system combining multiple NER systems’ outputs.The authors contrast this with traditional ensemble learning and describe the span-prediction combiner as end-to-end, flexible, and sharing parameters with base-system optimization.Source: §1 · p. 2
• The authors implement 154 systems on 11 datasets and report that their span prediction combiner outperforms several typical ensemble algorithms.The evaluation is described as comprehensive.Source: §1 · p. 2
• The authors provide an online demo and integrate their method into the NER Leaderboard to help researchers identify complementary system combinations and search for a new state-of-the-art system.Source: §1 · p. 2

2.1 Task

• NER is formulated as assigning each input token an output label, with F1-score used as the evaluation metric.The input and output are sequences of tokens and labels, respectively.Source: §2.1 · p. 2

2.2 Datasets

• The datasets cover multiple languages and domains, including English CoNLL-2003, Dutch and Spanish CoNLL-2002, and English OntoNotes genres.Although CoNLL-2003 covers English and German, this study uses only its English dataset; English OntoNotes domains are used to test robustness.Source: §2.2 · p. 2
• WNUT-2016 and WNUT-2017 provide Twitter social media data from shared tasks.Source: §2.2 · p. 3

3 Span Prediction for NE Recognition

• The section explores how SPANNER design choices affect its performance and interprets complementary strengths between SPANNER and SEQLAB.Source: §3 · p. 3

3.1 SPANNER as NER System

• SPANNER enumerates sentence spans for labeling and comprises token-representation, span-representation, and span-prediction modules.Source: §3.1 · p. 3, §3.1.2 · p. 3

3.1.1 Token Representation Layer

• The token layer applies pretrained embeddings, such as GloVe or BERT, followed by a bidirectional LSTM.Source: §3.1.1 · p. 3

3.1.2 Span Representation Layer

• Each span is represented using its start and end token embeddings, optionally augmented with a learnable span-length embedding.Source: §3.1.2 · p. 3

3.1.3 Span Prediction Layer

• The prediction layer scores span-label compatibility and uses heuristic decoding to retain the highest-probability span among overlaps in flat NER.Source: §3.1.3 · p. 3

3.2 Exp-I: Effectiveness of Model Variants

• Combining heuristic decoding with span-length features significantly improves SPANNER over the generic model on all datasets.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 Exp-II: Analysis of Complementarity

• SEQLAB leads on long, low-consistency entities and long, low-OOV sentences; SPANNER leads on medium entities and higher-OOV sentences, while feature-enhanced SPANNER improves on long sentences and entities.Source: Table 2 · p. 4, §3.3 · p. 5

• SPANNER serves both as a base model identifying potential entities and as a meta-model scoring each potential entity.Source: §4 · p. 5
• For each test span, P(s, l) is the combined probability of assigning label l, based on labels predicted by the base systems.Source: §4 · p. 5
• For “New York,” SPANNER’s LOC prediction joins three base-system predictions—LOC, ORG, and O—in the combination.The figure caption describes score(New York, LOC) = 0.5 as a span-label matching score.Source: §4 · p. 5, Figure 2 · p. 5
• The combiner sums the weights of base systems predicting each label and selects the label with the maximum score.Source: §4 · p. 5

5.1 Base Systems

• The 12 base systems comprised ten varied SEQLAB models plus two SPANNERs; no official EMLo language model was implemented for Spanish or Dutch.The SEQLAB designs varied character/subword representations, word representations, sentence encoders, and decoders.Source: §5.1 · pp. 5–6, Table 3 · p. 6

5.2 Baselines

• Six competing system-combination methods covered both supervised and unsupervised approaches.The evaluation included voting and stacking methods.Source: §5.2 · p. 6

5.2.1 Voting-based Approaches

• Voting baselines included majority voting and weights based on overall or class F1 scores.Overall-F1 weights came from testing-set scores; class-F1 weights were category-specific.Source: §5.2.1 · p. 6

5.2.2 Stacking-based Approaches

• Stacking compared SVM, Random Forest, and XGBoost combiners, requiring additional training samples that were collected through five-fold cross-validation.The authors characterize these methods as computationally expensive relative to SPANNER because they require external training samples.Source: §5.2.2 · pp. 6–7

5.3 Exp-III: Nuanced View

• Across 23 rank-based combination cases, SPANNER significantly beat six baselines in most cases, including “all”; adding systems improved combination performance.The best performance was always achieved with more models rather than only a small number of top-scoring systems.Source: Table 4 · p. 7, §5.3 · p. 7

5.4 Exp-IV: Aggregated View

• SPANNER achieved the best average and lowest standard deviation against three voting combiners on seven of nine datasets, with significance at p<0.05.The performance gap was larger on the web-domain WB and Twitter datasets W16 and W17.Source: Table 5 · p. 8, §5.4 · p. 8

5.5 Exp-VI: Interpretable Analysis

• Against base systems and combiners, SPANNER’s gains mainly involved low-label-consistency entities; SEQLAB bases benefited from short entities, SPANNER bases from long ones, while traditional combiners beat SPANNER on long sentences.The diagnosis uses F1 differences across entity attributes.Source: Table 6 · p. 8, §5.5 · p. 8

NER as Different Tasks

• NER work has shifted from token-level sequence labeling toward span prediction and reading-comprehension formulations (Ouchi et al., 2020).The paper focuses on complementarity between sequence labeling and span prediction.Source: §6 · p. 8

System Combination

• System combination has been studied in statistical and neural machine translation, while NER work has explored stacking, voting, and weighted voting.The cited NER approaches commonly require training samples and rely heavily on feature engineering.Source: §6 · pp. 8–9

Co-evolution of NLP Systems and their combiners

• The shared span-prediction formulation is proposed as a way to co-evolve NLP systems and combiners, with possible application to other NLP tasks such as constituency parsing and reranking.The paper describes formulating parsing and reranking as span-prediction problems parameterized in the same form.Source: §7 · p. 9

CombinaBoard

• COMBINABOARD lets researchers explore system complementarities, making a lower-ranked model’s value visible through combined results; the model is also deployed in ExplainaBoard for interactive combinations.The deployment supports flexible combination of top-scoring systems.Source: §7 · p. 9

A Attribute Interval

• Table 7 specifies XS, S, L, and XL intervals for eCon, sLen, eLen, and oDen.The passage names the attributes and bucket labels but does not provide their interval values.Source: Table 7 · p. 12, A Attribute Interval · p. 12
• Table 8 uses “Sub,” “sent.,” and “Dec.” for subword embedding, sentence encoder, and decoder; all ten SEQLAB models use CRF decoders.A check mark indicates an embedding or structure is used in the SEQLAB system.Source: Table 8 · p. 12

B Model Name illustration of SEQLAB

• All SEQLAB models use a CRF decoder.The examples sq0 and sq3 both include CRF in their full model names.Source: B Model Name illustration of SEQLAB · p. 12
• sq0 uses Flair character-level and GloVe word-level embeddings with an LSTM sentence encoder and CRF decoder.Its full model name is “CflairWglove lstmCrf”.Source: B Model Name illustration of SEQLAB · p. 12
• sq3 uses BERT as its character-level embedding, an LSTM sentence encoder, and a CRF decoder.Its full model name is “CbertWnon lstmCrf”.Source: B Model Name illustration of SEQLAB · pp. 12–13