PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, Jiawei Han

arXiv:1909.01441v1cs.CL

TL;DR

Annotation mistakes can interfere with NER evaluation and harm model training. The paper corrects CoNLL03 test annotations and proposes CrossWeigh, which detects potential training-label mistakes and reweights instances; experiments demonstrate its effectiveness across three datasets.

  • Problem

    Annotation mistakes in NER data can interfere with evaluation and hurt model training.

    Source: §1 · p. 1

  • Method

    CrossWeigh cross-checks training sentences with fold-trained models to flag potential mistakes, then reweights flagged instances for final-model training.

    Source: §3.3 · p. 5, §3.4 · p. 5

  • Results

    CrossWeigh's effectiveness was demonstrated on three datasets, and 5.38% of CoNLL03 test sentences contained label mistakes.

    Source: §1 · p. 1, §7 · p. 9

  • Takeaways & Limitations

    The findings indicate the potential of using CrossWeigh to improve annotation quality during label curation.

    Source: §7 · p. 9

    Iterative CrossWeigh is constrained by the efficiency of training multiple deep neural models hundreds of times.

    Source: §7 · p. 9

Abstract

from arXiv · show

Everyone makes mistakes. So do human annotators when curating labels for named entity recognition (NER). Such label mistakes might hurt model training and interfere model comparison. In this study, we dive deep into one of the widely-adopted NER benchmark datasets, CoNLL03 NER. We are able to identify label mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%. Therefore, we manually correct these label mistakes and form a cleaner test set. Our re-evaluation of popular models on this corrected test set leads to more accurate assessments, compared to those on the original test set. More importantly, we propose a simple yet effective framework, CrossWeigh, to handle label mistakes during NER model training. Specifically, it partitions the training data into several folds and train independent NER models to identify potential mistakes in each fold. Then it adjusts the weights of training data accordingly to train the final NER model. Extensive experiments demonstrate significant improvements of plugging various NER models into our proposed framework on three datasets. All implementations and corrected test set are available at our Github repo: https://github.com/ZihanWangKi/CrossWeigh.

1 Introduction

The paper examines how annotation mistakes affect NER evaluation and training, pairing a corrected CoNLL03 test set with CrossWeigh for training. Experiments report robust test F1 improvements when NER models use CrossWeigh across three datasets.

1 Introduction

  • Test-set mistakes can distort model evaluation, while training-set mistakes can hurt NER model training.Source: §1 · p. 1
  • 5.38% of CoNLL03 test sentences contained label mistakes; expert correction produced a cleaner set for more accurate and stable model re-evaluation.The authors report that the corrected test set better reflects model performance.Source: §1 · p. 1
  • CrossWeigh identifies potential training-label mistakes through cross-checking, then lowers the weights of flagged instances for final-model training.The cross-check uses models trained on folds that exclude entities appearing in the evaluated fold.Source: §1 · p. 2
  • CrossWeigh yields significant, robust test F1 improvements when plugged into NER models across three datasets.The reported datasets include CoNLL03, emerging-entity, and low-resource NER datasets.Source: §1 · p. 2

2 CoNLL03 NER Re-Examination

The CoNLL03 re-examination corrects test-set annotations and compares popular NER models on the corrected set. The authors report higher F1 scores and smaller standard deviations for all evaluated models on the corrected set than on the original.

2 CoNLL03 NER Re-Examination

  • CoNLL03 is divided into training, development, and test sets containing 14,041, 3,250, and 3,453 sentences, respectively.Its guideline marks person, location, organization, and miscellaneous entity types.Source: §2 · p. 2

2.1 Test Set Correction

  • Annotators checked each test sentence twice, followed by verification of disagreements; they corrected 186 sentences, or 5.38% of the test set.Five annotators shared the sentences, and reported inter-annotator agreement was 95.66%.Source: §2.1 · p. 3
  • Corrections applied the annotation guideline to entity types and used paragraph context to resolve a vessel reference as MISC.Source: §2.1 · p. 3

2.2 CoNLL03 Re-Evaluation

  • All evaluated models had higher F1 scores and smaller standard deviations on the corrected than original test set; LSTM-CRF averaged lower than LSTM-CNNs-CRF on the corrected set.Results are based on five runs.Source: Table 2 · p. 3, §2.2 · pp. 3–4

3 Our Framework: CrossWeigh

CrossWeigh accommodates imperfect NER training labels by estimating likely mistakes and reducing their influence in weighted training. Its estimation uses cross-checking that withholds entities appearing in each fold, while repeated estimates determine the degree of reweighting.

3 Our Framework: CrossWeigh

  • CrossWeigh is designed to fit most NER models, with the requirement that they can consume weighted training data.Source: §3 · p. 4

3.1 Overview

  • CrossWeigh estimates potential label mistakes through cross-checking, then lowers the weights of flagged instances during final NER training.The framework automates handling training-label mistakes that can hurt model performance.Source: §3.1 · p. 4

3.2 Preliminary

  • The weighted loss scales each sentence's model-versus-label loss by its sentence weight; ordinary NER training typically uses uniform weights of 1.Source: §3.2 · p. 4

3.3 Mistake Estimation

  • CrossWeigh trains a model for each fold after excluding training sentences containing entities from that fold, then flags sentences whose predicted labels disagree with their original labels.The entity-disjoint filtering is intended to test predictions without prior training information about those entities; repeated random partitions yield a mistake-confidence count from 0 to t.Source: §3.3 · p. 5

3.4 Mistake Reweighing

  • Flagged sentences receive weights that decay exponentially with their mistake-confidence count, while unflagged sentences retain weight 1.The parameter ε is chosen as 1 minus the estimated precision of detected mistakes, reflecting the fraction of flagged examples that may still be useful for training.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate CrossWeigh on CoNLL03 and other NER datasets, using multiple base models and parameter settings. They assess overall performance, mistake identification, and the effects of design choices.

4.1 Experimental Settings

  • Experiments use standard CoNLL03 splits, train on both train and development data, and report entity-wise test F1 averaged over five runs.The default CrossWeigh settings are k = 10, t = 3, and ϵ = 0.7; the base algorithm is mainly Flair, while Pooled-Flair is excluded from extensive experiments for computational cost.Source: §4.1 · p. 6

4.2 Overall Performance

  • CrossWeigh consistently raises F1 and yields comparable or sometimes smaller standard deviations with Flair, Pooled-Flair, and VanillaNER.The authors interpret the smaller standard deviations as greater training stability.Source: §4.2 · p. 6

4.3 Ablations and Variants

  • Without entity-disjoint filtering, F1 is close to raw Flair; filtering is reported critical for reducing overfitting and more effective than random discard.Across the tested c_i heuristics, performance differs little, while the default Ratio choice is reported as most stable.Source: §4.3 · p. 6, Table 4 · p. 6, Table 5 · p. 7

4.4 Label Mistake Identification Results

  • With Flair, CrossWeigh identifies more than 75% of test-set label mistakes at about 25% precision.The experiment applies the noise-estimation module to concatenated training and test data and uses manually corrected test labels to count true mistakes; the authors call 25% precision high given the 5.38% mistake ratio.Source: §4.4 · pp. 6–7, Table 9 · p. 8

4.5 Parameter Study

  • t = 3 is reported to provide a good-enough result, while larger t can stabilize estimation at greater computational cost.The study compares different iteration counts.Source: §4.5 · p. 7, Table 6 · p. 7
  • k = 5 and k = 10 perform significantly better than k = 2, with more folds trading higher computational expense for more training data per fold.The reported fold training sets grow from about 5,000 sentences and 1,500 entities at k = 2, to 7,000 and 4,000 at k = 5, and 9,000 and 7,000 at k = 10.Source: §4.5 · p. 7, Table 7 · p. 7
  • ϵ = 0.3 gives the worst performance, while ϵ = 0.5 performs on par with ϵ = 0.7 and is slightly better on the original test set.The authors hypothesize that ambiguous sentences omitted from their estimation of mistake-identification quality may make actual precision higher.Source: §4.5 · p. 7, Table 8 · p. 7

4.6 Other Datasets

  • On WNUT’17 and Sinhalese, CrossWeigh yields significantly higher F1 and smaller standard deviations than training without it.The evaluation uses Flair for WNUT’17 and BERT followed by BiLSTM-CRF for Sinhalese, with the same CrossWeigh settings as the CoNLL03 experiments.Source: §4.6 · pp. 7–8, Table 10 · p. 8

5 Case Studies

The case studies illustrate how annotation choices and training-label mistakes affect the dataset and model predictions. CrossWeigh flags a training error and the resulting Flair model predicts the entity correctly across runs.

Test Set Correction

  • The correction retained recurring sports-team LOC/ORG labels because the cases were consistent and generally followed the annotation guideline.Examples include away-team names labeled LOC in “MINNESOTA AT MILWAUKEE” and “NEW YORK AT CALIFORNIA.”Source: §5 · p. 8

CrossWeigh Framework

  • CrossWeigh flagged the incomplete “Maccabi Tel Aviv” training annotation in all three estimation iterations, after which Flair with CrossWeigh predicted the full team as ORG in all five runs.The original Flair model failed to predict the test-set mention as ORG in all five runs, despite a similar “ORG number ORG number” training pattern.Source: §5 · p. 8

6 Related Work

Related work covers automatic mistake detection, cross-validation and boosting, and neural NER. The paper distinguishes its gold-standard NER setting and its down-weighting approach from these prior methods.

6.1 Mistake Identification

  • Earlier automatic label-error methods addressed POS tagging and were not directly applicable to NER, while Rehbein and Ruppenhofer (2017) studied machine-generated silver-standard data rather than gold-standard data.The POS-tagging examples include Nakagawa and Matsumoto (2002) and Helgadóttir et al. (2014); the latter corrected the Icelandic Frequency Dictionary.Source: §6.1 · p. 8

6.2 Cross Validation & Boosting

  • CrossWeigh’s mistake estimation resembles cross-validation but adds entity-disjoint filtering, which experiments identify as crucial; unlike boosting, it decreases weights for sentences flagged by that process.The paper contrasts this with boosting’s assumption of perfect training data and increased weights for instances a current model fails to classify.Source: §6.2 · p. 9

6.3 NER Algorithms

7 Conclusion & Future work

The paper corrects CoNLL03 label mistakes, re-evaluates recent NER models on the corrected test set, and proposes CrossWeigh to detect training-label mistakes and train a more robust model. Experiments show CrossWeigh is effective on three datasets, while extending it iteratively remains future work.

7 Conclusion & Future work

  • The paper corrects CoNLL03 label mistakes, re-evaluates recent NER models on the corrected test set, and proposes CrossWeigh to detect training-label mistakes and train a more robust model.Source: §7 · p. 9
  • CrossWeigh is effective on three datasets and may improve annotation quality during label curation.The authors plan an iterative extension, with meta-learning as a possible way to address the efficiency bottleneck of training multiple deep neural models hundreds of times.Source: §7 · p. 9
  • An iterative extension, similar to boosting algorithms, is future work; its bottleneck is the efficiency of training multiple deep neural models hundreds of times.The authors propose meta-learning as one possible solution.Source: §7 · p. 9
Loading 1909.01441v1…