PaperlayerView exact revision on arXiv

Source-linked AI summary

TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

Christoph Alt, Aleksandra Gabryszak, Leonhard Hennig

arXiv:2004.14855v1cs.CL

TL;DR

TACRED’s high error rate raises whether the benchmark has reached a performance ceiling and how annotation, data, and models contribute. The paper validates its hardest examples, revises the dataset, and analyzes model errors, finding that labeling is a major error source and that ambiguous relations and unmasked-entity heuristics drive many remaining errors.

  • Problem

    TACRED’s nearly 30% error rate leaves unclear whether performance can improve and how annotations, dataset design, and models contribute.

  • Method

    The paper validates 5K challenging development and test examples, revises TACRED, categorizes misclassifications, and verifies error hypotheses across three state-of-the-art models.

  • Results

    8% absolute F1 test error comes from labeling, while two ambiguous-relation groups account for most remaining errors and models exploit cues when entities are unmasked.

  • Takeaways & Limitations

    Carefully relabeled evaluation splits and difficulty-aware analysis can improve the accuracy and reliability of future relation-extraction evaluations.

  • Takeaways & Limitations

    The analysis is limited to TACRED, and crowdsourcing may be insufficient for producing high-quality development and test evaluation data.

Abstract

from arXiv · show

TACRED (Zhang et al., 2017) is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pre-training and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

1 Introduction

The paper revisits TACRED by validating its most challenging examples, producing a revised evaluation set and a taxonomy of common RE errors. It then tests these error hypotheses on state-of-the-art models, identifying ambiguous relations and unmasked-entity cues as major sources of errors.

  • Motivation: TACRED contains more than 106k crowd-annotated examples and is one of the largest and most widely used datasets for evaluating relation extraction methods.Relation extraction supports knowledge-base population and question answering, making benchmark quality consequential.
  • Data validation: The authors rank development and test examples using misclassifications from 49 RE models and manually validate the top 5k most challenging instances.This procedure concentrates linguistic annotation effort on the examples most difficult for existing models.
  • Data validation: The validation reveals widespread crowd-label errors and yields an extensively relabeled TACRED development and test set intended to improve evaluation accuracy and reliability.The revised dataset is provided as a patch to the original TACRED because of licensing restrictions.
  • Error analysis: The authors develop 9 linguistically motivated categories for common RE errors by manually analyzing challenging revised-test predictions.They formulate these categories as hypotheses and test them on the full test set through adversarial rewriting.
  • Findings: Experiments on three state-of-the-art RE models show that two groups of ambiguous relations cause most remaining errors, while models exploit dataset cues when entities are unmasked.The analysis is limited to TACRED, but the authors state that the approach applies to other RE datasets and release analysis code.

2 The TACRED Dataset

TACRED is a large, fully supervised, sentence-level binary relation extraction dataset organized across training, development, and test years. Its crowdsourced labels show high sampled correctness but only moderate inter-annotator agreement.

  • Dataset composition: 106k sentences form TACRED, with entity mention pairs from TAC KBP evaluations and 41 person- and organization-oriented relation types.Training uses 2009–2012, development uses 2013, and testing uses 2014.
  • Annotation process: All relation labels were crowdsourced through Amazon Mechanical Turk, with workers choosing type-compatible suggestions or assigning no relation.Head and tail entity mentions were highlighted in each example.
  • Data quality: 93.3% of 300 randomly sampled examples were manually validated as correct, while crowd-worker agreement was moderate at κ = 0.54 across 761 mention pairs.The correctness estimate comes from Zhang et al. (2017).

3 An Analysis of TACRED Label Errors

Validation of model-challenging TACRED examples revealed extensive crowd-label errors, especially for positive relations, while control examples were rarely revised. Correcting these labels substantially improved model performance and showed that annotation quality is a major contributor to TACRED errors.

  • Validation results: 49.9% of Challenging Test examples and 52.1% of Challenging Dev examples were relabeled by annotators.Revision rates for originally positive examples were 47.3% (Test) and 49.1% (Dev). Approximately 57% of negative examples received a positive relation label.
  • Validation results: 8.9% of Test and 8.1% of Dev Control examples were revised, far below the Challenging-group rates.The Challenging group contained approximately one-third negative examples, compared with a dataset average of 79.5%, suggesting greater difficulty with positive examples.
  • Validation results: κTest = 0.87 and κDev = 0.80 between annotators indicated high validation quality, whereas agreement with crowd labels was κTest = 0.55 and κDev = 0.53.Agreement with crowdsourced labels was lowest for Challenging examples, reaching κTest = 0.44.
  • Error types: The frequently erroneous crowd labels were per:cities of residence, org:alternate names, and per:other family.Observed errors included assigning a positive relation when the sentence did not express it and failing to assign a positive relation when it did.
  • Impact on model performance: 70.1% was the average model F1 on the revised Test split, an 8% improvement over 62.1% on the original split and a 21.1% error reduction.The results indicate that crowdsourced annotation quality is a major contributor to TACRED model error.

4 An Analysis of Model Errors

The section analyzes model misclassifications by developing linguistically motivated error categories, converting them into dataset-wide testable hypotheses, and evaluating them with a baseline and three state-of-the-art relation extraction models.

  • 4.1 Linguistic error analysis: Manual analysis of misclassified revised Control and Challenging test instances develops categories based on contextual and entity-specific features that may mislead models.The analysis focuses on instances misclassified by the majority of 49 models and considers features such as entity-type errors and distracting context.
  • 4.2 Testable error hypotheses: The categories become testable hypotheses by defining dataset-wide groups of examples satisfying suspected conditions, such as the presence of a same-type distractor entity.The hypotheses are tested through adversarial rewriting, including removing contextual entities and observing whether predictions change.
  • 4.3 Automatic analysis: The study evaluates its error hypotheses on a single-layer CNN baseline and three recent state-of-the-art relation extraction models.The baseline uses max-pooling and 300-dimensional GloVe embeddings, while the state-of-the-art models use pre-trained language models fine-tuned for relation extraction.
  • 4.3 Automatic analysis: The evaluated models were excluded from the 49-model set used to select challenging instances, avoiding bias in the automatic evaluation.This separation ensures that hypothesis evaluation does not reuse models involved in challenge-instance selection.

5 Model Error and Dataset Analysis

The analysis separates model errors into relation-argument errors and context misinterpretations, finding that context misinterpretation dominates misclassifications. Automatic analysis further identifies ambiguous relation groups and shows that models differ in how they use contextual and argument information.

  • 5.1 Manual misclassification analysis: Misclassification categories distinguish relation-argument errors from context misinterpretation, including inverted arguments, wrong arguments, and no-relation predictions.Relation-argument errors concern incorrectly assigned entity spans or types, while context misinterpretation concerns the sentential context of the arguments.
  • 5.1 Manual misclassification analysis: 43.3% of cases disagreed with the re-annotated ground truth, while annotator agreement on error categories was κTest = 0.83.Agreement was κTest = 0.67 when the No Relation category was excluded.
  • 5.1 Manual misclassification analysis: 95.8% of false predictions were caused by context misinterpretation, compared with 4.2% caused by argument errors.No-relation errors accounted for 63.6% of misclassified instances, while inverted or wrong argument assignments accounted for 13.2%.
  • 5.2 Automatic analysis: KnowBERT had the lowest full-test error rate at 7.9%, whereas the masked CNN model had the highest at 11.9%.All models showed similar group-level patterns, with above-average error rates for positive, same nertag&positive, and per:loc.
  • 5.2 Automatic analysis: The per:loc and same nertag&positive groups were the most ambiguous, and KnowBERT and SpanBERT appeared to identify complementary information.For per:loc, 78.6% of examples were predicted correctly by both models and 21.4% by only one model; for same nertag&positive, 12.8% were predicted by only one.
  • 5.2 Automatic analysis: 61.9 F1 was SpanBERT’s score on the difficulty-weighted test set, a 16% decrease from the unweighted revised test set.The weighting emphasized instances misclassified by more baseline models and effectively ignored instances correctly classified by all 49 models.

6 Related Work

Prior TACRED relation-extraction work includes recurrent, attention-based and graph-convolutional models, while newer approaches use several forms of pre-training. Related research also evaluates dataset difficulty and bias, and analyzes trained models using feature attribution, probing, and instance rewriting.

  • Relation Extraction on TACRED: PA-LSTM combines recurrence and attention, whereas GCN leverages graph convolutional neural networks for relation extraction on TACRED.
  • Relation Extraction on TACRED: Current relation-extraction approaches use token- or span-level pre-training, knowledge-enhanced entity-linked representations, or “matching the blanks” pre-training.
  • Dataset Evaluation: Researchers assess dataset difficulty through model results and study dataset bias and shallow heuristics in reading comprehension, sentiment analysis, natural language inference, and argument reasoning comprehension.
  • Analyzing trained Models: Trained-model analyses use occlusion or gradient-based feature relevance, probing tasks, and instance rewriting, with rewriting used here to test explanations rather than robustness.

7 Conclusion and Future Work

The paper concludes that TACRED’s evaluation is substantially affected by labeling errors and that models exploit heuristics when entities are unmasked. It calls for more careful dataset-split evaluation and metrics that account for instance difficulty.

  • 8% absolute F1 error on the test set was attributed to labeling errors after validating 5k challenging development and test examples.The findings identify labeling as a major error source in the TACRED relation extraction task.
  • Careful evaluation of development and test splits is needed when creating datasets through crowdsourcing.The conclusion emphasizes evaluation accuracy and reliability for future relation extraction datasets.
  • Models adopt heuristics when entities are unmasked, motivating evaluation metrics that consider an instance’s difficulty.The proposed direction combines robustness to unmasked-entity heuristics with difficulty-aware evaluation.

A Appendix · A.1 Hyperparameters

The appendix specifies optimizer, learning-rate schedules, dropout, and architecture settings for CNN, LSTM/Bi-LSTM, GCN, and self-attention models. These configurations largely reuse prior work while defining model-specific training and structural hyperparameters.

  • A.1 Hyperparameters: CNN training follows Zhang et al. (2017), using Adagrad with initial learning rate 0.1 for 50 epochs and 0.9 decay from epoch 15.The CNN uses 500 filters with sizes [2, 3, 4, 5].
  • A.1 Hyperparameters: CNN variants with BERT use initial learning rate 0.01, reduce it by 0.9 when validation F1 plateaus, and use 200 filters of sizes [2, 3, 4, 5].
  • A.1 Hyperparameters: The BiLSTM has two layers with hidden dimension 500 for each direction, while ELMo and BERT training decays learning rate by 0.9 when validation F1 plateaus.
  • A.1 Hyperparameters: GCN settings follow Zhang et al. (2018), using SGD with initial learning rate 0.3 and 0.9 reduction whenever validation F1 plateaus.The model uses dropout of 0.5, word dropout of 0.04, embedding and encoder dropout of 0.5, and path-centric pruning with K=1.
  • A.1 Hyperparameters: The GCN uses two 200-dimensional GCN layers and two 200-dimensional feedforward layers with ReLU activation.
  • A.1 Hyperparameters: Self-attention performs best with 8 layers, each using 8 attention heads, attention dropout of 0.1, 256-dimensional keys and values, and a 512-dimensional feedforward layer.Training uses Adam with initial learning rate 10−4 and 0.9 reduction whenever validation F1 plateaus.

A.2 Relation Extraction Performance

This section reports relation extraction performance for 49 models on TACRED and its revised version, using the entity-masking strategy of Zhang et al. (2017). It also distinguishes models that augment word embeddings with part-of-speech and named-entity information.

  • A.2 Relation Extraction Performance: Models with w/synt/sem add named-entity and part-of-speech embeddings to the input word embeddings.For models w/ POS/NER, these embeddings are concatenated to each input token embedding.
  • A.2 Relation Extraction Performance: Test performance is reported for all 49 evaluated models on both TACRED and the revised dataset.The results appear in Table 6.
  • A.2 Relation Extraction Performance: The evaluation uses the same entity-masking strategy as Zhang et al. (2017), replacing each entity with a special <NER>-{SUBJ, OBJ} token.The <NER> tag corresponds to the entity’s named-entity recognition tag.
Loading 2004.14855v1…