Source-linked AI summary
Span-based Joint Entity and Relation Extraction with Transformer Pre-training
Markus Eberts, Adrian Ulges
TL;DR
Joint entity and relation extraction requires identifying entities and the relations between them, while many existing systems address these steps separately or use token tagging. SpERT uses BERT with span-based candidate search, localized marker-free context, and strong negative sampling; it outperforms prior work by up to 2.6% relation-extraction F1. Its reported errors include incorrect entity spans and relations that are plausible from entity types but unsupported by the sentence.
Problem
Joint extraction must identify entities and relations together, although common methods often tackle the two subproblems separately.
Method
SpERT uses BERT embeddings with span-based entity and relation candidates, lightweight classifiers, localized marker-free context, and strong within-sentence negative sampling.
Results
2.6% relation-extraction F1 is the maximum reported improvement over prior work, with SpERT outperforming the state of the art across the evaluated datasets.
Takeaways & Limitations
Strong negative sampling, span filtering, and localized context make searching over all spans in an input sentence feasible.
Takeaways & Limitations
Reported errors include slightly incorrect entity spans and relations predicted from entity types despite lacking sentence-level support.
Abstract
from arXiv · showhide
We introduce SpERT, an attention model for span-based joint entity and relation extraction. Our key contribution is a light-weight reasoning on BERT embeddings, which features entity recognition and filtering, as well as relation classification with a localized, marker-free context representation. The model is trained using strong within-sentence negative samples, which are efficiently extracted in a single BERT pass. These aspects facilitate a search over all spans in the sentence. In ablation studies, we demonstrate the benefits of pre-training, strong negative sampling and localized context. Our model outperforms prior work by up to 2.6% F1 score on several datasets for joint entity and relation extraction.
1 INTRODUCTION
The paper presents SpERT, a BERT-based span model that jointly extracts entities and relations by searching over span hypotheses. It combines lightweight reasoning, localized context, and strong within-sentence negative sampling, outperforming prior work by up to 2.6% relation-extraction F1.
- SpERT jointly performs entity recognition and relation classification using BERT as its core Transformer network.
- Any token subsequence can be an entity candidate, and relations can hold between any pair of spans.This enables a full search over span hypotheses rather than token-level BIO/BILOU tagging.
- A single BERT forward pass produces embeddings for lightweight downstream entity and relation classifiers.The model uses localized, marker-free context and shallow processing after encoding.
- 2.6% relation-extraction F1 is the reported maximum improvement over prior work.The paper also reports that strong same-sentence negative samples, localized context, and fine-tuning pre-trained models improve performance.
2 RELATED WORK
Prior joint extraction systems commonly use sequence tagging, table filling, stacked models, or span representations built with recurrent encoders. SpERT instead uses an attention-based Transformer and a span-based formulation without manually defined relation questions.
- Many earlier joint models detect entities with BIO or BILOU sequence tags and then predict relations using recurrent or structured architectures.
- Table-filling methods represent token pairs in a sentence table, combining diagonal entity tags with off-diagonal relation types.Beam search can be used to find an optimal table-filling solution.
- SpERT uses an attention-based Transformer instead of the LSTM-heavy architectures common in prior approaches.
- Unlike a BERT-based question-answering approach, SpERT requires no manually defined relation-specific question templates and uses spans instead of BILOU tags.
- Span-based approaches can cover overlapping entities that single-token BIO/BILOU tags cannot represent simultaneously.
- Earlier span-based joint systems derived shared span representations from BiLSTM encoders over ELMo, word, and character embeddings.DyGIE added graph propagation, while DyGIE++ replaced the BiLSTM encoder with BERT.
3 APPROACH
SpERT uses BERT embeddings with span-based entity classification and filtering, followed by relation classification over entity pairs and localized context. Joint supervised training combines entity and relation losses while reusing one BERT pass per sentence.
- Span classification: BERT produces token and sentence-context embeddings that support classification of arbitrary candidate spans.The input is tokenized into BPE tokens, passed through BERT, and represented by embeddings plus a classifier token for overall sentence context.
- Span classification: Each span combines fused BERT embeddings, a learned width embedding, and the classifier token before softmax entity classification.The span classifier maps candidates to entity categories or none; max-pooling was found to work best for embedding fusion.
- Span filtering: Spans assigned to none are filtered, and spans longer than 10 tokens are excluded, limiting span classification cost to O(n).The remaining spans form the candidate entity set S for relation classification.
- Relation classification: The relation classifier evaluates ordered pairs from S×S using entity representations and context drawn from the span between the entities.Both pair orderings are classified because relations are generally asymmetric.
- Relation classification: A sigmoid relation classifier activates relations scoring at least α; if none reaches the threshold, the pair is assigned no known relation.The classifier outputs one response for each predefined relation class.
- Training: Supervised training jointly optimizes span and relation classification while fine-tuning BERT, using cross-entropy for spans and binary cross-entropy for relations.Training samples are drawn per sentence, allowing multiple examples to share a single BERT forward pass and substantially speeding training.
4 EXPERIMENTS
SpERT is evaluated on three joint entity and relation extraction datasets and consistently outperforms prior systems. Ablations show that sufficient strong negatives, localized context, BERT pre-training, and max-pooled span representations materially improve performance.
- 4.1 Comparison with state of the art: SpERT consistently outperforms state-of-the-art systems for entity and relation extraction across CoNLL04, SciERC, and ADE.Entity F1 improves by 1.1% on CoNLL04 and 2.8% on SciERC; relation extraction gains are larger.
- 4.1 Comparison with state of the art: 2.6% micro F1 improvement is achieved over Multi-turn QA for relation extraction on CoNLL04.On SciERC, SpERT outperforms DyGIE++ by about 2.4% using SciBERT.
- 4.2 Candidate selection and negative sampling: Sufficient negative samples are essential: relation F1 is about 10.5% on CoNLL04 and 9.7% on SciERC with one negative entity and relation per sentence.Performance stagnates at high sample counts, while Ne=Nr=100 produces more stable results in other experiments.
- 4.2 Candidate selection and negative sampling: Weak negative relation samples retain 84.4% recall but reduce precision to about 4.3%, encouraging predictions involving entity subspans.Filtering entity candidates before sampling strong negatives is described as vital for accurate localization.
- 4.3 Localized context: Localized context reaches 71.0% F1, exceeding whole-sentence max pooling at 65.8% and classifier-token context at 63.9%.For sentences longer than 50 tokens, localized context reaches 57.3% F1 versus 44.9/38.5% for the other options.
- 4.4 Pre-training and entity representation: Removing BERT layer pre-training reduces entity and relation macro F1 by about 17.0% and 29.4%, respectively.Training token embeddings from scratch causes an even stronger F1 decline, indicating that language-model pre-training is vital for competitive performance.
5 CONCLUSIONS
SpERT makes exhaustive span-based joint entity and relation extraction feasible with BERT through strong negative sampling, span filtering, and localized context. The authors identify richer context modeling as a key direction for future work.
- 5 CONCLUSIONS: SpERT combines BERT with strong negative sampling, span filtering, and localized context to make exhaustive search over sentence spans feasible.The model searches over possible entity spans and relations while retaining an efficient BERT-based design.
- 5 CONCLUSIONS: Span-based approaches perform competitively with BILOU-based models and can identify overlapping entities.The authors describe this capability as a reason span-based approaches may be promising for future research.
- 5 CONCLUSIONS: Future work should investigate more elaborate relation-classifier context, syntactic features, and learned context while preserving efficient exhaustive search.The current model uses the span between two entities, which outperformed full-sentence context.