Source-linked AI summary
Re-TACRED: Addressing Shortcomings of the TACRED Dataset
George Stoica, Emmanouil Antonios Platanios, Barnabás Póczos
TL;DR
TACRED’s widespread use made annotation reliability important, yet prior quality evidence came from a small, biased sample. The paper re-annotates the full dataset with an improved crowdsourcing strategy, finding substantial label corrections and a 14.3% average f1-score improvement while releasing Re-TACRED for evaluation.
Problem
Prior TACRED quality analysis used a small, biased sample, limiting generalization about annotation reliability across the full dataset.
Method
The paper comprehensively reviews and re-annotates TACRED using an improved crowdsourcing strategy with refined definitions and quality assurance.
Results
14.3% average f1-score improvement was observed for models evaluated on the revised dataset.
Takeaways & Limitations
Re-TACRED provides revised labels for more reliable evaluation of relation extraction models.
Takeaways & Limitations
Handling wrong entity types with separate annotation tasks could increase worst-case annotation cost by approximately 130%.
Abstract
from arXiv · showhide
TACRED is one of the largest and most widely used sentence-level relation extraction datasets. Proposed models that are evaluated using this dataset consistently set new state-of-the-art performance. However, they still exhibit large error rates despite leveraging external knowledge and unsupervised pretraining on large text corpora. A recent study suggested that this may be due to poor dataset quality. The study observed that over 50% of the most challenging sentences from the development and test sets are incorrectly labeled and account for an average drop of 8% f1-score in model performance. However, this study was limited to a small biased sample of 5k (out of a total of 106k) sentences, substantially restricting the generalizability and broader implications of its findings. In this paper, we address these shortcomings by: (i) performing a comprehensive study over the whole TACRED dataset, (ii) proposing an improved crowdsourcing strategy and deploying it to re-annotate the whole dataset, and (iii) performing a thorough analysis to understand how correcting the TACRED annotations affects previously published results. After verification, we observed that 23.9% of TACRED labels are incorrect. Moreover, evaluating several models on our revised dataset yields an average f1-score improvement of 14.3% and helps uncover significant relationships between the different models (rather than simply offsetting or scaling their scores by a constant factor). Finally, aside from our analysis we also release Re-TACRED, a new completely re-annotated version of the TACRED dataset that can be used to perform reliable evaluation of relation extraction models.
1 Introduction
The paper re-annotates the full TACRED dataset to address shortcomings in its original data collection and annotation quality. It introduces a cost-efficient crowdsourcing strategy, analyzes label corrections, and releases Re-TACRED.
- Re-annotating the entire TACRED dataset addresses shortcomings that limited the reliability of prior evaluations.
- The proposed crowdsourcing strategy targets flaws in data collection, refines relation definitions, and adds quality-assurance mechanisms.
- 14.3% average f1-score improvement followed correction of TACRED labels when evaluating existing relation extraction models.
- Re-TACRED is released publicly as a patch containing the revised labels because licensing restrictions prevent release of the complete dataset.
2 Background
TACRED is a large, widely used sentence-level relation extraction dataset whose annotation quality has been questioned. Earlier verification found substantial label problems, but its small biased sample limited generalization to the full dataset.
- TACRED contains over 106,000 sentences from knowledge-base population evaluations, split into training, development, and test periods.
- Each instance identifies subject and object spans with types, and relations are typed for specific subject-object combinations, including NO RELATION.
- Original annotations were collected through AMT multiple-choice labeling using relation suggestions compatible with the entity types.
- That revised sample improved average f1-score by 8%, but its small and biased coverage restricted conclusions about the full dataset.
3 TACRED Revision
The paper revises TACRED through a full-dataset crowdsourcing strategy that handles incorrect entity types, ambiguous relation definitions, and annotation quality. Its design combines super-clustered candidate labels, refined definitions, quality controls, and targeted dataset cleanup.
- 3.2 Wrong Type Handling: 5% of a 1,000-instance sample had incorrect subject or object types, making type-compatible candidate relations potentially unusable.The original task design did not allow annotators to select the correct relation when the supplied types were wrong.
- 3 TACRED Revision: The authors re-annotate the full TACRED dataset with an improved crowdsourcing design addressing data errors, label ambiguity, and annotation quality.The strategy is deployed through AMT rather than on only a small sample.
- 3.2 Wrong Type Handling: 8 super-clusters group confused sentence types and reduce worst-case annotation cost by 27/8 ≈3.4.Each cluster uses the union of its member groups’ candidate labels, improving the chance that incorrectly typed sentences retain a compatible option.
- 3.3 Relation Definition Refinements: The revision also extends the PERSON:ALTERNATE RELATION definition for pronominal mentions referring to the same person.This addresses inconsistencies involving PERSON:OTHER RELATION, which accounts for nearly 10% of TACRED.
- 3.3 Relation Definition Refinements: Relation definitions are refined to separate overlapping residence and birth cases and clarify organization-location relations.For example, mentions of “native” cannot receive a residence relation, while headquarters relations are generalized to include branches or offices.
- 3.4 Quality Assurance: A two-step quality process filters poor annotators before labeling and checks consistency during annotation.The stages are called the trial and the control.
4 TACRED and Re-TACRED Comparison
Re-TACRED changes a substantial share of TACRED annotations and materially reshapes relation distributions. The revisions expose systematic confusions, especially among residence, birth, and organization-location relations.
- 4.1 Qualitative Comparison: 23.9% of sentences receive labels that disagree with the original TACRED annotations.The revised dataset contains 91,467 sentences while preserving TACRED’s train, development, and test splits.
- 4.1 Qualitative Comparison: 41.8% more sentences receive PERSON:CITY BIRTH than in TACRED, and 55.2% of these were originally PERSON:CITIES RESIDENCE.This illustrates the boundary changes between the two relations.
- 4.1 Qualitative Comparison: 67.5% more organization-location labels are assigned, with 93.9% of those revisions originally labeled NO RELATION.The authors attribute this increase primarily to revised definitions and improved handling of subject and object types.
- 4.1 Qualitative Comparison: PERSON:CITIES RESIDENCE has the largest reduction, with 44.6% of its sentences re-annotated differently.This complements the increased number of PERSON:CITY BIRTH labels.
4.2 Model Performance Comparison
Re-TACRED changes model comparisons substantially: performance improves across models and relation categories, while revealing stronger differences between systems than TACRED suggested.
- Overall Performance Impact: 15.6% f1-measure, 15.1% precision, and 16.2% recall are SpanBERT’s largest improvements between TACRED and Re-TACRED.All three models improve significantly across every metric, and the asymmetric changes indicate that revisions do not simply offset or scale scores.
- Overall Performance Impact: C-GCN outperforms PA-LSTM by as much as 1.7% in Precision on Re-TACRED, despite being only marginally better on TACRED.The revised dataset makes their performance difference more pronounced.
- Performance Change Across Label Types: C-GCN outscores PA-LSTM in every analyzed category on Re-TACRED, while SpanBERT consistently leads both models across both datasets.TACRED presents C-GCN and PA-LSTM as comparable, whereas Re-TACRED reveals C-GCN as substantially stronger.
- Effect of Refined Labels: 88.0% is the largest improvement from refined-label re-annotations, with SpanBERT averaging a 31.4% f1-score increase across categories.PERSON:IDENTITY shows SpanBERT’s largest category gain, increasing f1-measure by 71.4%.
- Effect of Non-Refined Labels: Re-TACRED-trained models achieve substantially more correct positive predictions than TACRED-trained models on TACRED test data.For example, SpanBERT trained on Re-TACRED has 49.4% fewer correct positive instances on TACRED_test than on Re-TACRED_test.
5 Conclusion
The paper re-annotates the complete TACRED dataset and evaluates how corrected labels change relation-extraction model comparisons. Across three models, the revised dataset produces an average 14.3% f1-score improvement.
- 5 Conclusion: The study re-annotates the complete TACRED dataset using crowdsourcing and examines the resulting changes in model evaluation.The review addresses data errors, label-definition ambiguity, and annotation quality control.
- 5 Conclusion: 14.3% is the average f1-score improvement across three models on the revised dataset.The authors analyze how existing relation-extraction methods compare between TACRED and Re-TACRED and how their errors change.
- 5 Conclusion: Re-TACRED changes the observed comparison and error patterns of existing relation-extraction methods between datasets.The conclusion frames these changes as a central outcome of the comparative analysis.
A Hyperparameters
The experiments largely retain the original models’ reported training settings while checking selected hyperparameter alternatives for Re-TACRED.
- A Hyperparameters: The models use their contributors’ reported hyperparameters for TACRED-based training.Training uses the original model configurations as the primary comparison settings.
- A Hyperparameters: PA-LSTM and SpanBERT each undergo limited grid searches, but their best Re-TACRED results use the original reported parameters.PA-LSTM searches LSTM dimensions, depth, dropout, and position encoding; SpanBERT searches learning rate and warm-up proportions.
- A Hyperparameters: Re-TACRED C-GCN retains the reported training procedure and changes only the residual network hidden dimension from 200 to 300.The authors report that most original hyperparameters yield the best results.
B Amazon Mechanical Turk
The annotation process uses multiple rounds of crowdsourced labeling with disagreement resolution and embedded quality checks.
- B Amazon Mechanical Turk: Each of eight annotation tasks uses two to four rounds, beginning with two annotations per sentence and escalating disagreements to additional workers.Third and fourth rounds address remaining disagreements when necessary.
- B Amazon Mechanical Turk: Each HIT contains five sentences, including one gold item, and costs $0.15.The gold sentence provides a known-label quality check within each task.
- B Amazon Mechanical Turk: 243 workers complete the annotation rounds over approximately 784 sequential hours, with parallel rounds reducing elapsed time.The reported labor total reflects sequential annotation time rather than wall-clock duration.
C Relation Alterations
The authors identify inconsistent labeling of identity-pronominal sentences and explain that ambiguity in TAC KBP guidelines motivates revising the relation scheme.
- C Relation Alterations: Identity-pronominal sentences show substantial label inconsistencies in TACRED.They were frequently assigned either PERSON:OTHER or another relation.
- C Relation Alterations: Ambiguous TAC KBP guidelines make the original relation definitions insufficient for identity-pronominal relationships.The authors motivate a dedicated PERSON:IDENTITY relation by examining the affected definitions and their shortcomings.
C.1 Definitions
The affected TAC KBP label definitions are presented in Table 7.
- C.1 Definitions: Table 7 presents the TAC KBP sentence definitions for the affected relations.
C.2 Encoding Identity Shortcomings
The original relation definitions leave pronominal identity ambiguous, so the authors introduce PERSON:IDENTITY and revise related labels to encode these cases explicitly.
- C.2 Encoding Identity Shortcomings: The original definitions do not explicitly cover identity sentences involving pronominal reference.PERSON:ALTERNATE-NAMES requires different names, while PERSON:OTHER-FAMILY could be interpreted inconsistently.
- C.2 Encoding Identity Shortcomings: Annotator interpretations could assign pronominal relationships to PERSON:OTHER-FAMILY or NO-RELATION.The original assignments therefore reflected subjective readings of whether pronominal relations counted as familial.
- C.2 Encoding Identity Shortcomings: The authors create PERSON:IDENTITY to explicitly cover pronominal relationships between sentence entities.This relation generalizes PERSON:ALTERNATE-NAMES, whose cases are absorbed into PERSON:IDENTITY.
- C.2 Encoding Identity Shortcomings: PERSON:OTHER-FAMILY is barred from accepting pronominal relationships, while other relation definitions remain unchanged.