PaperlayerView exact revision on arXiv

Source-linked AI summary

Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels

Elena Merdjanovska, Jonas Golde, Alan Akbik

arXiv:2608.30827v1cs.CL

TL;DR

LLM-generated labels can introduce heterogeneous NER noise that standard fine-tuning and uniform noise-robust losses do not handle well. The paper proposes error-type-aware loss reweighting that masks potentially unreliable tokens by error type, improving F1 across noisy datasets without additional training resources. Its scope is limited by assumptions underlying the Beta-mixture thresholding procedure, which may yield degraded performance on datasets with different or smaller distributions.

  • Problem

    LLM-generated NER annotations can contain heterogeneous errors, so treating all noisy tokens uniformly may remove useful supervision or reinforce incorrect labels.

  • Method

    The method uses separate error-type-aware reweighting rules and confidence-based reliability masks, estimating thresholds from training data with a two-component Beta mixture.

  • Results

    0.8–2.0 percentage-point per-dataset test F1 improvements over standard cross-entropy are reported, with targeted error-type-aware masking more effective than uniform masking.

  • Takeaways & Limitations

    The efficient approach can be applied in a single fine-tuning run without additional resources for small domain-specific NER models trained on LLM-annotated data.

  • Takeaways & Limitations

    The method assumes errors can be modeled with two Beta components and a monotone posterior; on smaller or differently distributed datasets, thresholds may be unreliable and performance may degrade.

Abstract

from arXiv · show

Large language models are increasingly used to annotate datasets for training smaller, task-specialized models such as named entity recognition. While this method yields effective models, it assumes that the synthetic dataset is correctly annotated. In this work, we find that (i) current fine-tuning processes simply ignore LLM-introduced annotation noise, resulting in degraded performance and (ii) existing noise-robust losses are not transferable to sequence labeling because annotation noise in named entity recognition is heterogeneous: for example, missing mentions and type errors affect the training signal in different ways. Treating all noisy tokens equally in noise-robust losses and applying a single reweighing criterion for all may therefore remove useful supervision or reinforce incorrect labels. To address this limitation, we propose error-type-aware loss reweighting for NER, which introduces separate reweighing rules for different types of potentially erroneous tokens. Our approach is simple and efficient, does not require additional training resources, and improves F1 by 0.8 - 2.0 percentage points on dataset-level average for noise levels between 15% and 40%, with a maximum improvement of 4.6 percentage points with 24.1% noise on Wikigold.

1 Introduction

LLM-generated NER annotations can transfer noise into fine-tuned models, while heterogeneous error types make uniform noise handling unsuitable. The paper proposes error-type-aware loss reweighting and evaluates it across datasets, annotators, models, and noise levels.

  • Motivation: LLM-generated annotation noise creates a practical ceiling on downstream NER performance.On OntoNotes, fine-tuning on LLM-labeled data remained within 3 F1 points of prompting across four datasets.
  • Motivation: NER annotation errors include missing entities, hallucinated entities, wrong types, and incorrect boundaries.These errors are heterogeneous, and NER also has severe class imbalance because most tokens receive the O label.
  • Approach: The proposed method identifies potentially mislabeled tokens using disagreement between model predictions and annotations, combined with model confidence.It adapts which tokens to mask according to the error type present in the LLM-annotated data.
  • Contributions: The paper analyzes error profiles across datasets and prompting approaches, finding that they differ greatly even with the same model.It evaluates three LLM annotators across four datasets and two NER models against standard fine-tuning and noise-robust baselines.
  • Results: 0.8–2.0 percentage-point per-dataset F1 improvements over standard cross-entropy are obtained with DistilBERT, compared with 1.2–1.6 points with XLM-RoBERTa (large).The results also report that error-type-aware reweighting outperforms global token reweighting, with L_missing the best-performing loss overall.

2 Related Work

Prior noisy-NER methods largely target distant or weak supervision and often rely on extra training stages, voters, or clean data. This paper’s related-work context motivates direct error-type filtering without additional training stages or clean validation data.

  • Noisy Labels in NER: Existing noisy-NER research has focused extensively on distant supervision, using self-training, noisy-label removal, and noise-aware losses.Examples include BOND, RoSTER, and NEEDLE.
  • Reliability Estimation: Many methods estimate token reliability through model confidence or training dynamics, but several require multiple models, iterations, or retraining.These approaches filter positive and negative mention samples separately or use trained-model voters and cleaned-dataset retraining.
  • Comparison: Self-Cleaning reweights tokens affected by boundary and type errors separately using a discriminator trained on a small clean set.The proposed approach similarly filters by error type but applies the filtering directly in the loss without additional training stages or a clean validation set.
  • Setting: Distant- and weak-supervision methods that rely on clean development data degrade substantially when that data is noisy.The paper therefore adopts a setting without clean data, while recognizing that different annotation sources introduce different noise types, including LLM-generated labels.

3 Method

The method estimates token label reliability from model confidence and applies hard loss masking selectively. It uses separate masking rules for heterogeneous NER error types rather than one rule for all disagreements.

  • The method trains an NER model despite noisy token labels, where corruption probability varies by instance and observed label.
  • A binary reliability mask weights each token’s categorical cross-entropy loss, with zero weight completely excluding a token.
  • The posterior uses the token’s input, observed label, current model prediction, and confidence score to estimate its latent clean state.
  • A two-component Beta mixture fitted to training confidence scores identifies likely clean and noisy tokens without clean validation data.
  • Separate masking rules partition disagreements into missing-entity, entity, false-positive, and type-error categories, with a global-threshold variant as baseline.
  • The categorization is imperfect because tokens in missing-entity and entity sets can also belong to boundary errors.

4 Experimental Setup

The experiments train on noisy annotations and evaluate on clean gold-label test splits without using clean data for model selection. They cover four NER datasets, multiple LLM annotation sources, and additional noisy-label settings.

  • Models are trained on noisy annotations and evaluated on clean test splits with gold labels to isolate annotation-noise effects.
  • The no-clean-data setting uses a noisy validation split for model selection.
  • The evaluation covers NoiseBench, OntoNotes, Wikigold, and BC5CDR, spanning entity-type inventories from 2 to 18 labels.
  • Additional evaluations use distant-supervision labels and five further NoiseBench noisy variants.
  • LLM annotations come from two models and four prompting approaches, with three selected label variants per dataset.
  • The baselines include categorical cross-entropy, generalized cross-entropy, focal loss, BMM bootstrap loss, and corrected NLL.

5 Results

The results analyze heterogeneous LLM annotation errors, evaluate error-type-aware losses across datasets and models, and compare them with baselines and masking upper bounds. Error-specific masking generally improves performance, but combined masking can remove too much useful supervision.

  • 5.1 LLM Annotation Errors: LLM label variants show dataset- and prompt-dependent error profiles, including missing mentions, false positives, type errors, and partial boundary errors.BC5CDR is dominated by missing mentions, whereas NoiseBench prompts often produce many type errors; higher-quality variants for several datasets have no single error type above 40%.
  • 5.2 Evaluation of Error-Type-Aware Losses: Lmissing is best on all OntoNotes and BC5CDR variants, while Lentity is best on NoiseBench and LFP is best on average for Wikigold.On NoiseBench GPT-OSS Basic, Lentity raises F1 from 68.9 to 72.5; on Wikigold GPT-OSS EvoPrompt, LFP raises F1 from 60.2 to 64.8.
  • 5.2 Evaluation of Error-Type-Aware Losses: In 9 of 12 label variants, the best-performing loss targets the most frequent error category, although broader Lentity masking is more effective than further splitting entity errors.None of the existing noise-robust losses outperform the proposed variants on any dataset.
  • 5.3 Upper Bounds for Loss Masking: Ideal masking approaches show that excluding erroneous tokens can recover performance close to clean-data fine-tuning, while Ideal Lall provides a smaller but more realistic upper-bound improvement.Ideal Masking removes all erroneous tokens; Ideal Lall masks erroneous tokens only when the model misclassifies them, so its mask changes across epochs.
  • 5.5 Why Combined Masking Underperforms: Combined masking underperforms because it masks too many tokens, whereas Lmissing’s adaptive threshold initially masks many tokens and gradually masks fewer during training.The combined loss masks over twice as many tokens as Lmissing or Lentity separately, and its noisy-development F1 fails to improve.

6 Conclusion

The paper addresses noisy LLM-generated supervision for NER by targeting different error types separately. Across varied datasets and label variants, targeted masking improves test F1 and remains practical for fine-tuning small domain-specific models.

  • Variable LLM annotation quality can substantially affect downstream NER model performance.
  • The proposed loss reweighting approach targets different types of NER errors separately.
  • 0.8 to 2.0 percentage points: targeted error-type-aware masking improves test F1 per dataset over the evaluated noisy-label settings.The study evaluates four datasets, each with three LLM-generated label variants.
  • The method can be applied in a single fine-tuning run without additional resources.This supports fine-tuning small domain-specific NER models on LLM-annotated data.

Limitations

The method’s scope is bounded by modeling assumptions, dataset conditions, and evaluation coverage. The authors also report that no unified error mode was found and that combined masking degraded performance across investigated benchmarks.

  • No unified error mode was found, and combined masking approaches degraded performance across the investigated benchmarks.
  • The method assumes error types are explained by two Beta components with a posterior monotone in score, neither of which is guaranteed.
  • Fewer examples, fewer classes, or different distributions may produce unreliable thresholds and degraded performance.
  • The experiments cover only DistilBERT and XLM-RoBERTa on English data, without testing autoregressive models or low-resource languages.
  • The evaluation is limited to token-level NER and does not consider span-level classification.

A.1 Datasets

The experiments use established datasets and selected noisy LLM-label variants produced with multiple prompts, annotator models, and refinement procedures. Fine-tuning evaluates two encoder models under specified training and masking configurations.

  • Datasets: Wikigold is split into train, development, and test sets using an 80% / 10% / 10% split because it lacks an established split.
  • Datasets: NoiseBench, BC5CDR, and OntoNotes retain their original train-test splits.
  • Prompts and annotators: The study uses Basic prompts for all datasets and Schema prompts for Wikigold and NoiseBench, with EvoPrompt and DiZiNER adaptations.
  • Prompts and annotators: Two annotator models produce 4–6 label variants per dataset, from which three representative variants are selected across noise levels.
  • Training: DistilBERT and XLM-RoBERTa (large) are fine-tuned for 10 epochs with three random seeds.
  • Loss configuration: Error-type-aware losses use a 150-iteration warmup, posterior cutoff 0.3, and threshold updates after epochs 1, 3, and 6.Thresholds are initialized at 0.4 for Lmissing and 0.85 for Lentity, using a fitted BMM.
  • Loss configuration: An entity-coverage guard masks tokens only after the model predicts at least 10% of observed non-O tokens in a batch.The guard prevents training collapse when no positive non-O examples remain.

B LLM Error Propagation

The analysis compares fine-tuned-model errors with LLM annotation errors to measure error propagation. Missing mentions and partial matches are more strongly inherited from LLM errors than type and hallucination errors.

  • The analysis compares direct LLM test annotations with predictions from models fine-tuned on corresponding LLM-labeled training data.
  • Around 80% of missing-mention and partial-match errors occur at LLM error locations.
  • 55% of type errors and 63% of hallucinated errors occur at LLM error locations.
  • Fine-tuning largely inherits LLM missing and span-boundary errors, while wrong types and hallucinations are less strongly associated with LLM errors.

C Results with XLM-RoBERTa (large)

The section presents main results tables using XLM-RoBERTa (large), whereas the main paper’s results otherwise use DistilBERT.

  • XLM-RoBERTa (large) is used for the section’s alternative-model results, while Figure 1 is the only exception elsewhere in the main paper.

D Results on Extended Noisy Variants

The section evaluates loss variants on extended noisy-label variants and noise-free data using both DistilBERT and XLM-RoBERTa (large). It compares error-type-aware losses with cross-entropy, global masking, and related noise-robust losses.

  • The evaluation covers extended noisy variants and a dataset without noise.
  • Table 7 reports test F1 for error-type-aware losses on LLM label variants using XLM-RoBERTa (large).The comparison includes cross-entropy and global masking.
  • Table 8 compares error-aware losses with other related noise-robust losses using test F1 and XLM-RoBERTa (large).
  • Tables 9 and 10 report test F1 for error-type-aware losses against cross-entropy and global masking on extended label variants.Table 9 uses DistilBERT, while Table 10 uses XLM-RoBERTa (large).
Loading 2608.30827v1…