Source-linked AI summary
An Improved Baseline for Sentence-level Relation Extraction
Wenxuan Zhou, Muhao Chen
TL;DR
Sentence-level relation extraction is affected by incomplete entity representations and noisy or ill-defined labels. The paper proposes an improved baseline using typed entity markers within a PLM-based classifier. It reports 74.6% F1 on TACRED and 91.1% F1 on refined Re-TACRED, showing stronger performance than earlier results.
Problem
Existing RE models inadequately represent entity information, while noisy or ill-defined labels can misestimate performance on datasets such as TACRED.
Method
The paper introduces typed entity marker representations and integrates them into a PLM-based sentence-level RE baseline.
Results
74.6% F1 on TACRED and 91.1% F1 on Re-TACRED are achieved, with the TACRED result significantly outperforming previous state-of-the-art RE models.
Takeaways & Limitations
The results indicate that PLMs can achieve much better and satisfactory performance on sentence-level RE when entity representation and label quality are addressed.
Abstract
from arXiv · showhide
Sentence-level relation extraction (RE) aims at identifying the relationship between two entities in a sentence. Many efforts have been devoted to this problem, while the best performing methods are still far from perfect. In this paper, we revisit two problems that affect the performance of existing RE models, namely entity representation and noisy or ill-defined labels. Our improved RE baseline, incorporated with entity representations with typed markers, achieves an F1 of 74.6% on TACRED, significantly outperforms previous SOTA methods. Furthermore, the presented new baseline achieves an F1 of 91.1% on the refined Re-TACRED dataset, demonstrating that the pretrained language models (PLMs) achieve high performance on this task. We release our code to the community for future research.
1 Introduction
Sentence-level relation extraction remains imperfect despite extensive PLM-based research. The paper identifies incomplete entity representation and noisy or ill-defined labels as key obstacles, then proposes an improved baseline with typed entity markers and evaluates it across TACRED variants.
- Relation extraction predicts the relationship between two entities from a predefined set of relationships.
- Existing sentence-level RE models use external knowledge or relation-oriented pretraining, yet performance remains far from perfect.
- Incomplete representation of entity names, spans, and types limits entity characterization in RE models.
- Noisy or ill-defined human labels in datasets such as TACRED can cause model performance to be misestimated.
- The proposed improved baseline introduces typed entity markers for sentence-level RE.
- 74.6% F1 on TACRED and 83.2% on TACREV significantly outperform various state-of-the-art RE models.
2 Method
The method represents sentence-level RE inputs with entity spans and types, processes them with a pretrained language model, and classifies the entity-pair relation. Its central representation is a typed marker scheme that preserves entity text while exposing subject and object types.
- Problem Definition: Sentence-level RE predicts a relation for a subject-object entity pair from R ∪{NA}, assigning NA when no target relation is expressed.
- Model Architecture: The classifier marks entity spans and types, encodes the processed sentence with a PLM, and uses subject and object hidden states for relation prediction.
- Entity Representation: Entity names, spans, and NER types provide clues to relation types and must be represented in the input text for PLMs to capture them.
- Entity Representation: Entity markers enclose subject and object spans with special token pairs, while punctuation markers use @ and # without adding reserved-vocabulary tokens.
- Entity Representation: Typed entity markers add subject and object NER types to the enclosing markers using type-specific special tokens.
- Entity Representation: Typed entity marker (punct) marks spans with @ and # and prepends textual subject and object types enclosed by * and ∧.
3 Experiments
Experiments evaluate entity representations and the improved baseline across TACRED variants, using pretrained encoders and comparisons with prior methods. Typed entity markers perform strongly, and the resulting RoBERTa-based baseline achieves new state-of-the-art results across all datasets.
- Experimental Setup: Experiments use TACRED, TACREV, and Re-TACRED, with BERTBASE, BERTLARGE, and RoBERTaLARGE encoders for representation analysis.The study also compares against previously proposed RE systems and reports median F1 over five runs.
- Entity Representation Analysis: Typed entity markers outperform untyped representations, with RoBERTa achieving 74.6% F1 using the typed marker (punct) on TACRED.The result exceeds LUKE's 72.7% SOTA result on TACRED.
- Entity Representation Analysis: Entity-marker symbols affect performance even when the represented entity-information categories remain the same.The analysis distinguishes original and punct marker variants.
- Comparison with Prior Methods: The improved baseline uses RoBERTaLARGE with the typed entity marker (punct) and achieves new SOTA results on TACRED, TACREV, and Re-TACRED.The baseline is compared with prior methods in the test-set evaluation.
- Comparison with Prior Methods: On Re-TACRED, the typed-marker gain is smaller than on TACRED and TACREV, decreasing to 0.2−0.8% F1 from 3.1−3.9% and 2.0−3.4%.The paper attributes this pattern to noisy TACRED labels biased toward entity side information.
- Additional Analyses: Additional analyses examine generalization to unseen entities and performance under annotation errors.These analyses are reported in the appendix.
4 Conclusion
The paper presents a simple, strong relation-extraction baseline and studies entity representation and noisy or ill-defined labels. Its techniques produce new state-of-the-art performance, including 91.1% F1 on Re-TACRED.
- Conclusion: The paper presents a simple, strong RE baseline with a comprehensive study of prediction generalizability and robustness.The study revisits entity representation and noisy or ill-defined labels.
- Conclusion: The proposed entity representation technique significantly outperforms existing sentence-level RE models.The conclusion identifies entity representation as a central technical problem addressed by the paper.
- Conclusion: 91.1% F1 on Re-TACRED shows that PLMs already achieve satisfactory performance on this task.The paper presents this result as evidence from its improved baseline.
A Dataset Statistics
The experiments use three TACRED dataset versions, whose statistics are reported in Table 3. TACREV and Re-TACRED reflect successive relabeling and refinement of TACRED annotations.
- Dataset Statistics: Table 3 reports statistics for the datasets used in the experiments.The referenced dataset-statistics table covers the TACRED variants used in evaluation.
B Analysis on Unseen Entities
The unseen-entity analysis filters test sets to instances whose entities never appear in training and compares entity masks with typed entity markers. Typed markers consistently perform better, indicating that entity names can generalize to unseen entities.
- Motivation: Prior work questioned whether entity names leak superficial relation clues and whether models without entity masks generalize to unseen entities.These concerns motivate the filtered evaluation.
- Filtered Evaluation: Filtered test sets remove instances containing training-set entities, leaving 4,599 instances for TACRED and TACREV and 3815 for Re-TACRED.The remaining instances contain entities unseen during training.
- Evaluation Design: Results compare entity-mask and typed-marker representations across encoders and datasets, but original and filtered test results are not directly comparable.The label distributions differ between the two test-set types.
- Results: Typed entity markers consistently outperform entity masks on all encoders and datasets.This result supports learning from entity names in the unseen-entity setting.
- Results: The findings indicate that entity names provide semantically richer information than entity types and can support generalization to unseen entities.The paper connects this result with prior findings from Peng et al. (2020).
C Analysis on Annotation Errors
The analysis attributes performance differences across TACRED variants largely to annotation errors, which can distort evaluation of entity-informed models. A cleaned TACRED test set helps isolate this effect.
- A clean TACRED test set was created by pruning instances whose annotated relation differs in the Re-TACRED test set.The remaining instances are treated as clean for analysis.
- The model gains less on Re-TACRED than on TACRED and TACREV, with the difference mainly attributed to evaluation-set annotation errors.
- TACRED and TACREV annotation errors can overestimate the performance of models using entity side information.The authors hypothesize that some annotators labeled relations from the entity pair without reading the full sentence.
- The authors characterize TACRED and TACREV evaluation sets as biased and unreliable, recommending Re-TACRED for future sentence-level RE evaluation.