Source-linked AI summary
Enriching Pre-trained Language Model with Entity Information for Relation Classification
Shanchan Wu, Yifan He
TL;DR
Relation classification requires modeling both a sentence and its two target entities, a setting not previously addressed with pretrained BERT in this paper’s account. The paper marks entity locations with special tokens and combines sentence and entity representations for classification, achieving a new state of the art on SemEval-2010 Task 8. Its scope is regular relation classification rather than distantly supervised data with noisy labels.
Problem
Relation classification depends on information from both the whole sentence and the specific target entities, while pretrained BERT had not been applied to this task in the paper’s account.
Method
R-BERT inserts special tokens around each target entity and combines the resulting entity representations with the sentence vector for classification.
Results
R-BERT achieves a new state of the art for relation classification, with a MACRO F1 of 89.25 on the SemEval-2010 benchmark dataset.
Takeaways & Limitations
Enriching pretrained BERT with entity-level information captures sentence semantics and target-entity information for relation classification.
Takeaways & Limitations
The study focuses on regular relation classification and does not address distantly supervised data with noisy labels.
Abstract
from arXiv · showhide
Relation classification is an important NLP task to extract relations between entities. The state-of-the-art methods for relation classification are primarily based on Convolutional or Recurrent Neural Networks. Recently, the pre-trained BERT model achieves very successful results in many NLP classification / sequence labeling tasks. Relation classification differs from those tasks in that it relies on information of both the sentence and the two target entities. In this paper, we propose a model that both leverages the pre-trained BERT language model and incorporates information from the target entities to tackle the relation classification task. We locate the target entities and transfer the information through the pre-trained architecture and incorporate the corresponding encoding of the two entities. We achieve significant improvement over the state-of-the-art method on the SemEval-2010 task 8 relational dataset.
1 Introduction
Relation classification predicts semantic relations between nominal pairs, but differs from typical BERT tasks because it requires both sentence-level and target-entity information. The paper addresses this gap by enriching BERT with entity representations and reports a new state of the art.
- Task motivation: Relation classification predicts semantic relations between pairs of nominals and serves as an intermediate step in various NLP applications.The task identifies the relation between two target nominals given a text sequence, usually a sentence.
- Prior approaches: Deep neural relation-classification methods commonly derive features from lexical resources and NLP tools such as dependency parsers and named entity recognizers.
- Prior approaches: BERT achieved state-of-the-art results on eleven NLP tasks, which were typically formulated as classification or sequence-labeling problems.BERT was also applied to question answering by finding answer-span boundaries.
- Proposed approach: The proposed model inserts special tokens around both target entities, then combines their output embeddings with the sentence encoding for classification.This design transfers entity-location information through BERT while retaining sentence semantics.
- Contribution: The paper introduces an approach incorporating entity-level information into a pretrained language model for relation classification and claims a new state of the art.
2 Related Work
Prior relation-classification work used deep neural architectures, while distant-supervision approaches face noisy labels. This paper focuses on regular relation classification without noisy labels.
- Deep learning methods: Earlier relation-classification methods included recursive neural networks and convolutional neural networks using syntactic structure, word embeddings, and position features.The MVRNN builds sentence representations bottom up from parse-tree structure, while the CNN combines word and position features.
- Distant supervision: Distant-supervision research addresses relation extraction using automatically generated data that may contain a large number of noisy labels.
- Scope: This paper focuses on regular relation classification rather than distantly supervised data with noisy labels.
3 Methodology
R-BERT adapts pre-trained BERT for relation classification by marking two target entities and combining sentence-level and entity-level representations. The resulting representation is passed through fully connected and softmax layers to predict relation probabilities.
- 3.1 Pre-trained Model BERT: R-BERT uses pre-trained BERT, whose token representations combine token, segment, and position embeddings.BERT is a multi-layer bidirectional Transformer encoder.
- 3.2 Model Architecture: Special '$' and '#' tokens mark the beginnings and ends of the two target entities before BERT processing.A '[CLS]' token is also added at the beginning of each sentence.
- 3.2 Model Architecture: The model averages BERT hidden states within each target entity, then applies tanh and a fully connected layer to obtain two entity vectors.The entity vectors are denoted H′1 and H′2.
- 3.2 Model Architecture: The '[CLS]' hidden state receives its own tanh activation and fully connected transformation.The model shares parameters between the two entity-processing transformations.
- 3.2 Model Architecture: The transformed sentence and entity representations are concatenated, then processed by a fully connected layer and softmax to produce relation probabilities.The output layer uses W3 ∈ R^L×3d, where L is the number of relation types.
- 3.2 Model Architecture: R-BERT uses cross entropy loss and applies dropout before each fully connected layer during training.The paper calls this approach R-BERT.
4 Experiments
Experiments evaluate R-BERT on SemEval-2010 Task 8 against established baselines and through component ablations. R-BERT achieves the strongest reported performance, while removing entity markers or entity representations reduces performance.
- Dataset and evaluation: The SemEval-2010 Task 8 dataset contains nine semantic relation types, one Other class, and 10,717 sentences.Evaluation uses macro-averaged F1 over the nine actual relations, excluding Other, while considering directionality.
- Comparison with other methods: R-BERT is compared with SVM, RNN, MVRNN, CNN+Softmax, FCM, CR-CNN, Attention-CNN, and Entity Attention Bi-LSTM.The comparison includes the previous best solution from the SemEval-2010 Task 8 competition.
- Comparison with other methods: 89.25 MACRO F1 is reported for R-BERT, significantly beating all baseline methods and surpassing the previous best solution.The result is reported for the SemEval-2010 Task 8 dataset.
- Effect of model components: BERT-NO-SEP-NO-ENT performs worst among the ablations, with F1 8.16 absolute points below R-BERT.This configuration removes both the special entity-separation tokens and the hidden entity vectors used for classification.
- Effect of model components: All three ablated methods perform worse than R-BERT, indicating contributions from both special separation tokens and hidden entity vectors.The tokens identify target-entity locations and transfer that information into BERT, while entity vectors enrich the classification representation.
5 Conclusions
The paper enriches pre-trained BERT with target-entity information for relation classification. On SemEval-2010, the approach significantly outperforms state-of-the-art methods; extending it to distant supervision is proposed as future work.
- Conclusion: The approach enriches pre-trained BERT with entity information by combining sentence and target-entity representations for relation classification.Special tokens are added around each target entity pair.
- Conclusion: Experiments on SemEval-2010 show that the proposed results significantly outperform state-of-the-art methods.The paper identifies extending the model to distant supervision as possible future work.