Re-TACRED: Addressing Shortcomings of the TACRED Dataset

• Re-annotating the entire TACRED dataset addresses shortcomings that limited the reliability of prior evaluations.Source: §1 · p. 1
• The proposed crowdsourcing strategy targets flaws in data collection, refines relation definitions, and adds quality-assurance mechanisms.Source: §1 · pp. 1–2
• 14.3% average f1-score improvement followed correction of TACRED labels when evaluating existing relation extraction models.Source: §1 · p. 2
• Re-TACRED is released publicly as a patch containing the revised labels because licensing restrictions prevent release of the complete dataset.Source: §1 · p. 2

• TACRED contains over 106,000 sentences from knowledge-base population evaluations, split into training, development, and test periods.Source: §2 · p. 2
• Each instance identifies subject and object spans with types, and relations are typed for specific subject-object combinations, including NO RELATION.Source: §2 · p. 2
• Original annotations were collected through AMT multiple-choice labeling using relation suggestions compatible with the entity types.Source: §2 · p. 2
• That revised sample improved average f1-score by 8%, but its small and biased coverage restricted conclusions about the full dataset.Source: §2.1 · p. 2

3.2 Wrong Type Handling

• 5% of a 1,000-instance sample had incorrect subject or object types, making type-compatible candidate relations potentially unusable.The original task design did not allow annotators to select the correct relation when the supplied types were wrong.Source: §3 · p. 3

3 TACRED Revision

• The authors re-annotate the full TACRED dataset with an improved crowdsourcing design addressing data errors, label ambiguity, and annotation quality.The strategy is deployed through AMT rather than on only a small sample.Source: §3 · p. 3

3.2 Wrong Type Handling

• 8 super-clusters group confused sentence types and reduce worst-case annotation cost by 27/8 ≈3.4.Each cluster uses the union of its member groups’ candidate labels, improving the chance that incorrectly typed sentences retain a compatible option.Source: §3.2 · p. 3

3.3 Relation Definition Refinements

• The revision also extends the PERSON:ALTERNATE RELATION definition for pronominal mentions referring to the same person.This addresses inconsistencies involving PERSON:OTHER RELATION, which accounts for nearly 10% of TACRED.Source: §3.3 · p. 4
• Relation definitions are refined to separate overlapping residence and birth cases and clarify organization-location relations.For example, mentions of “native” cannot receive a residence relation, while headquarters relations are generalized to include branches or offices.Source: §3.3 · p. 4

3.4 Quality Assurance

• A two-step quality process filters poor annotators before labeling and checks consistency during annotation.The stages are called the trial and the control.Source: §3.4 · p. 5

4.1 Qualitative Comparison

• 23.9% of sentences receive labels that disagree with the original TACRED annotations.The revised dataset contains 91,467 sentences while preserving TACRED’s train, development, and test splits.Source: §4.1 · p. 5, §4 · p. 5
• 41.8% more sentences receive PERSON:CITY BIRTH than in TACRED, and 55.2% of these were originally PERSON:CITIES RESIDENCE.This illustrates the boundary changes between the two relations.Source: §4.1 · p. 5
• 67.5% more organization-location labels are assigned, with 93.9% of those revisions originally labeled NO RELATION.The authors attribute this increase primarily to revised definitions and improved handling of subject and object types.Source: §4.1 · p. 5
• PERSON:CITIES RESIDENCE has the largest reduction, with 44.6% of its sentences re-annotated differently.This complements the increased number of PERSON:CITY BIRTH labels.Source: §4.1 · p. 6

Overall Performance Impact

• 15.6% f1-measure, 15.1% precision, and 16.2% recall are SpanBERT’s largest improvements between TACRED and Re-TACRED.All three models improve significantly across every metric, and the asymmetric changes indicate that revisions do not simply offset or scale scores.Source: SpanBERT · p. 6, §4.2 · p. 6, Table 2 · p. 6
• C-GCN outperforms PA-LSTM by as much as 1.7% in Precision on Re-TACRED, despite being only marginally better on TACRED.The revised dataset makes their performance difference more pronounced.Source: SpanBERT · p. 6

Performance Change Across Label Types

• C-GCN outscores PA-LSTM in every analyzed category on Re-TACRED, while SpanBERT consistently leads both models across both datasets.TACRED presents C-GCN and PA-LSTM as comparable, whereas Re-TACRED reveals C-GCN as substantially stronger.Source: SpanBERT · p. 6, Table 3 · p. 7

Effect of Refined Labels

• 88.0% is the largest improvement from refined-label re-annotations, with SpanBERT averaging a 31.4% f1-score increase across categories.PERSON:IDENTITY shows SpanBERT’s largest category gain, increasing f1-measure by 71.4%.Source: Dataset · p. 7, Table 4 · p. 7

Effect of Non-Refined Labels

• Re-TACRED-trained models achieve substantially more correct positive predictions than TACRED-trained models on TACRED test data.For example, SpanBERT trained on Re-TACRED has 49.4% fewer correct positive instances on TACRED_test than on Re-TACRED_test.Source: Dataset · pp. 7–8

5 Conclusion

• The study re-annotates the complete TACRED dataset using crowdsourcing and examines the resulting changes in model evaluation.The review addresses data errors, label-definition ambiguity, and annotation quality control.Source: §5 · pp. 8–9
• 14.3% is the average f1-score improvement across three models on the revised dataset.The authors analyze how existing relation-extraction methods compare between TACRED and Re-TACRED and how their errors change.Source: §5 · pp. 8–9
• Re-TACRED changes the observed comparison and error patterns of existing relation-extraction methods between datasets.The conclusion frames these changes as a central outcome of the comparative analysis.Source: §5 · p. 9

A Hyperparameters

• The models use their contributors’ reported hyperparameters for TACRED-based training.Training uses the original model configurations as the primary comparison settings.Source: A Hyperparameters · p. 10
• PA-LSTM and SpanBERT each undergo limited grid searches, but their best Re-TACRED results use the original reported parameters.PA-LSTM searches LSTM dimensions, depth, dropout, and position encoding; SpanBERT searches learning rate and warm-up proportions.Source: A Hyperparameters · p. 10
• Re-TACRED C-GCN retains the reported training procedure and changes only the residual network hidden dimension from 200 to 300.The authors report that most original hyperparameters yield the best results.Source: A Hyperparameters · p. 10

B Amazon Mechanical Turk

• Each of eight annotation tasks uses two to four rounds, beginning with two annotations per sentence and escalating disagreements to additional workers.Third and fourth rounds address remaining disagreements when necessary.Source: B Amazon Mechanical Turk · p. 10
• Each HIT contains five sentences, including one gold item, and costs $0.15.The gold sentence provides a known-label quality check within each task.Source: B Amazon Mechanical Turk · p. 10
• 243 workers complete the annotation rounds over approximately 784 sequential hours, with parallel rounds reducing elapsed time.The reported labor total reflects sequential annotation time rather than wall-clock duration.Source: B Amazon Mechanical Turk · p. 10

C Relation Alterations

• Identity-pronominal sentences show substantial label inconsistencies in TACRED.They were frequently assigned either PERSON:OTHER or another relation.Source: C Relation Alterations · p. 10
• Ambiguous TAC KBP guidelines make the original relation definitions insufficient for identity-pronominal relationships.The authors motivate a dedicated PERSON:IDENTITY relation by examining the affected definitions and their shortcomings.Source: C Relation Alterations · p. 10

C.1 Definitions

• Table 7 presents the TAC KBP sentence definitions for the affected relations.Source: §C.1 · p. 10

C.2 Encoding Identity Shortcomings

• The original definitions do not explicitly cover identity sentences involving pronominal reference.PERSON:ALTERNATE-NAMES requires different names, while PERSON:OTHER-FAMILY could be interpreted inconsistently.Source: §C.2 · p. 10
• Annotator interpretations could assign pronominal relationships to PERSON:OTHER-FAMILY or NO-RELATION.The original assignments therefore reflected subjective readings of whether pronominal relations counted as familial.Source: §C.2 · p. 10
• The authors create PERSON:IDENTITY to explicitly cover pronominal relationships between sentence entities.This relation generalizes PERSON:ALTERNATE-NAMES, whose cases are absorbed into PERSON:IDENTITY.Source: §C.2 · p. 10
• PERSON:OTHER-FAMILY is barred from accepting pronominal relationships, while other relation definitions remain unchanged.Source: §C.2 · p. 10