Source-linked AI summary
Simple BERT Models for Relation Extraction and Semantic Role Labeling
Peng Shi, Jimmy Lin
TL;DR
Relation extraction and semantic role labeling have commonly relied on lexical and syntactic features, motivating simpler approaches that avoid external resources. This paper builds BERT-based models for both tasks and evaluates them on benchmark datasets. The models achieve state-of-the-art results in several settings without linguistic features or human-designed constraints, providing strong baselines for future research.
Problem
Existing neural models for relation extraction and semantic role labeling typically rely on lexical and syntactic features, while parser availability and robustness are limited.
Method
The paper develops simple BERT-based architectures for relation extraction and semantic role labeling without external lexical or syntactic features.
Results
The models achieve state-of-the-art performance across benchmark datasets and several SRL settings without linguistic features or declarative decoding constraints.
Takeaways & Limitations
The models provide strong baselines and foundations for future research on these two tasks.
Abstract
from arXiv · showhide
We present simple BERT-based models for relation extraction and semantic role labeling. In recent years, state-of-the-art performance has been achieved using neural models by incorporating lexical and syntactic features such as part-of-speech tags and dependency trees. In this paper, extensive experiments on datasets for these two tasks show that without using any external features, a simple BERT-based model can achieve state-of-the-art performance. To our knowledge, we are the first to successfully apply BERT in this manner. Our models provide strong baselines for future research.
1 Introduction
Relation extraction and semantic role labeling are fundamental language-understanding tasks whose strongest neural models often use lexical and syntactic features. The paper asks whether BERT can deliver state-of-the-art performance without those external features and reports that simple BERT-based architectures do so.
- Relation extraction identifies relations between entity pairs, while semantic role labeling extracts predicate–argument structures.Both tasks support downstream applications including question answering and open information extraction.
- State-of-the-art neural models typically incorporate lexical and syntactic features, including part-of-speech tags, syntactic trees, and global decoding constraints.
- Parser dependence is limiting because parsers may be unavailable for some languages or insufficiently robust on out-of-domain text.Such parser weaknesses may even hurt performance.
- Simple neural architectures built on BERT achieve state-of-the-art performance across benchmark datasets for relation extraction and semantic role labeling without lexical or syntactic features.The paper presents this as an affirmative answer to whether pretrained models can replace those features.
2 BERT for Relation Extraction
The relation extraction model combines entity-aware BERT representations with relative position information, then uses a BiLSTM and MLP for prediction. On TACRED, it achieves the best known F1 among individual models while remaining below an interpolation model because of lower recall.
- Model: The model encodes relation extraction inputs as [CLS] sentence [SEP] subject [SEP] object [SEP].Entity mentions in the sentence are replaced with masks encoding argument type and entity type to prevent overfitting.
- Model: BERT contextual representations are augmented with position embeddings relative to the subject and object entity spans.The position values represent token distance from each entity span.
- Model: The concatenated representations pass through a one-layer BiLSTM, whose final directional hidden states feed a one-hidden-layer MLP for prediction.
- Experiments: The TACRED experiments use BERT base-cased, LSTM and MLP hidden sizes of 768 and 300, and position embeddings of size 20.The learning rate is 5 × 10^-5, and mask and position embeddings are fine-tuned.
- Experiments: The model obtains the best known F1 among individual models on TACRED, but remains below Zhang et al. (2018)'s interpolation model because of lower recall.It outperforms Zhang et al. (2018) and Wu et al. (2019), which use syntactic tree information, while Alt et al. (2019) achieves better recall.
3 BERT for Semantic Role Labeling
The paper formulates semantic role labeling around predicate disambiguation and argument identification/classification, using BERT-based models without linguistic features. Across dependency- and span-based benchmarks, the BERT-LSTM-large model achieves strong or state-of-the-art results, with some dataset-specific limitations.
- Task formulation: SRL is treated through predicate disambiguation plus argument identification and classification, covering both span-based and dependency-based annotation.The paper discusses CoNLL 2005, 2009, and 2012 benchmarks, while focusing on predicate disambiguation and argument processing because predicates are given in several benchmarks.
- Predicate sense disambiguation: The predicate disambiguation model casts the task as sequence labeling, marking the predicate with its sense label and other tokens with BERT-compatible O/X labels.A predicate indicator embedding is concatenated with BERT contextual representations before final prediction by a one-hidden-layer MLP.
- Argument identification and classification: The argument model encodes a sentence–predicate pair with BERT and predicts BIO-style argument-role tags for each token.Its predicate-aware input places the sentence and predicate together, allowing attention between the predicate and the sentence; Figure 2 describes combining predicate and current-token hidden states before an MLP.
- Dependency-based SRL results: The model achieves better predicate disambiguation accuracy than a feature-based BiLSTM state-of-the-art model and reports separate development, test, and Brown evaluations.These disambiguation results feed the dependency-based end-to-end evaluation.
- SRL results: The BERT-LSTM-large model yields large end-to-end F1 improvements over prior state of the art and beats existing ensembles without linguistic features or declarative decoding constraints.For span-based SRL, it reaches state-of-the-art F1 among single models and outperforms an ensemble on CoNLL 2005, but falls short on CoNLL 2012 because that ensemble achieves very high precision with complex constraints.
4 Conclusions
BERT can be adapted to relation extraction and semantic role labeling without syntactic features or human-designed constraints. The authors present this simplicity as a strength and position the models as strong baselines for future research.
- BERT can be adapted to relation extraction and semantic role labeling without syntactic features and human-designed constraints.
- The authors argue that model simplicity is beneficial because BERT can simplify architectures tailored to specific tasks.
- The results provide strong baselines and foundations for future research.
- The authors identify reintroducing syntactic features and using multitask learning as directions for follow-up research.