FLERT: Document-Level Features for Named Entity Recognition

• Earlier studies used document-level features alongside other contributions, leaving their impact in isolation unevaluated.Source: §1 · p. 1
• The comparison covers fine-tuning a transformer with a linear prediction layer and using transformer features in a standard LSTM-CRF without fine-tuning.Source: §1 · p. 1
• Document-level features significantly improve NER quality, fine-tuning generally outperforms feature-based approaches, and the best settings yield new CoNLL benchmark scores.Source: §1 · p. 1

• Self-attention over the input allows both left and right context to influence the target sentence’s representations.Source: §2 · p. 1, Figure 1 · p. 2
• For each target sentence, the method adds 64 subtokens of left and right context, rather than building context separately for each token.Source: §2 · p. 2
• Per-sentence context limits the added window, requires each sentence and its context to pass through the transformer only once, and permits sentence shuffling each training epoch.Source: §2 · p. 2

3 Baseline Parameter Experiments

• The best configurations from the parameter study are carried forward into the final comparative evaluation.Source: §3 · p. 2

3.1 Setup

• The study uses CoNLL development datasets for English, German, Dutch, and Spanish, including original and revised German, with XLM-R and optional classic word embeddings.Source: §3.1 · p. 2

3.2 First Approach: Fine-Tuning

• Fine-tuning updates the transformer and adds a linear prediction layer after pooling subtokens into token-level representations.Source: §3.2 · p. 2

3.3 Second Approach: Feature-Based

• The feature-based approach freezes the transformer and feeds its word embeddings to an LSTM-CRF; variants use either a mean across all layers or concatenated last-four-layer representations.Source: §3.3 · p. 3

3.4 Results: Best Configurations

• Additional word embeddings and a CRF decoder help fine-tuning only for some languages and often minimally, so the study selects Transformer-Linear; feature-based results favor all-layer mean with word embeddings.Source: §3.4 · p. 3

4.1 Main Results

• ≈↑2 pp on average: fine-tuning outperforms the feature-based approach, and document-level features beat sentence-level features by ↑1.15 pp on average.These comparisons use the best configurations on the CoNLL-03 test splits.Source: §4.1 · p. 3, §4 · p. 3, Table 3 · p. 4
• Context-window impact is marginal, with 64 tokens performing best across languages.The comparison evaluates windows of 48, 64, 96, and 128 tokens on development data.Source: Table 4 · p. 4, §4.2 · p. 4
• ORG and PER entities improve most with document-level features, indicating that cross-sentence context is especially important for these types.Source: Table 5 · p. 4, §4.2 · p. 4
• Document-boundary enforcement increases F1 across nearly all experiments, and fine-tuning with document-level features works best across languages.The authors recommend enforcing boundaries where possible; they report new state-of-the-art scores for German and Dutch, but not English or Spanish.Source: §4.1 · p. 4

• Document-level features significantly improve overall F1, and fine-tuning outperforms the feature-based LSTM-CRF.Source: §5 · p. 4
• Enforcing document boundaries improves results, potentially adding evidence that transformers have difficulty learning positional signals.Source: §5 · p. 4
• The FLERT extension integrates the approach into FLAIR to enable researchers to use the selected setups for NER models.Source: §5 · p. 4

• Fine-tuning adds a linear layer and tunes the entire transformer architecture on the NER task.Source: §A.1 · p. 7
• First-subword pooling uses the first subtoken's representation for each token before token-level predictions.This bridges subtoken modeling and token-level predictions.Source: §A.1 · p. 7
• The one-cycle schedule linearly decreases the learning rate to zero by the end of training.Prior works typically use AdamW, a very small learning rate, and a small fixed number of epochs as a stopping criterion.Source: §A.1 · p. 7

• Transformer features are input to an LSTM-CRF, using either the mean across all layers or the concatenated last four layers.Source: §A.2 · p. 7
• Feature-based training uses SGD with a larger learning rate annealed against development data, stopping when the learning rate becomes too small.Source: §A.2 · p. 7
• First-subword pooling represents each token using its first subtoken, including when a word is split into multiple subtokens.The illustration shows “Eiffel” split into three subtokens and represented by the first, “E”.Source: Figure 2 · p. 7, §A.2 · p. 7

A.3 Reproducibility Checklist

• Table 8 reports the number of sentences for each CoNLL dataset.The supplied passage identifies the table's scope but gives no sentence-count values.Source: §A.3 · p. 7, Table 8 · p. 7
• Table 9 reports average training runtimes for the approaches, measured on an NVIDIA V-100 for fine-tuning and an NVIDIA RTX 3090 TI for the feature-based approach.The reported runtimes are averages for the best configurations.Source: §A.3 · p. 7, Table 9 · p. 7
• The feature-based approach calculates self-attention over all input tokens, including left and right context, then represents sentence tokens by averaging all transformer layers or concatenating the last four.Figure 3 depicts the sentence “I love Paris” as the represented token span.Source: Figure 3 · p. 8
• XLM-R has 550M reported parameters; the fine-tuned model has 560M (↑1.8%) and the feature-based model has 564M (↑2.5%).The reported evaluation metric is the averaged F1-score over three runs, using the CoNLL-2003 evaluation script.Source: §A.3 · p. 8