Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels

• The method assumes most LLM labels are correct and uses model-prediction disagreement and confidence to identify potentially mislabeled tokens, adapting masking to the error type.Source: §1 · p. 2
• The study analyzes error patterns that vary across datasets and prompting approaches, and evaluates six loss variants across four datasets, three LLM annotators, and two models against standard fine-tuning and noise-robust baselines.Source: §1 · p. 2
• Per-dataset F1 improvements over standard cross-entropy are between 0.8 and 2.0 percentage points with DistilBERT and 1.2 - 1.6 with XLM-RoBERTa (large).Source: §1 · p. 2
• Error-type-aware reweighting outperforms global token reweighting, and Lmissing, which targets missing annotations, is the best-performing loss overall.Source: §1 · p. 2

Noisy Labels in NER

• BOND (Liang et al., 2020), RoSTER (Meng et al., 2021), and NEEDLE (Jiang et al., 2021) address distant or weak supervision with self-training, noisy-label removal, or a noise-aware loss.Source: §2 · p. 2
• Reliability methods use model confidence or training dynamics to identify unreliable tokens, but often require extra resources such as multiple models or retraining on cleaned data.Examples include Li et al. (2025), Liu et al. (2021), Zhang et al. (2025), and Merdjanovska and Akbik (2025).Source: §2 · pp. 2–3
• Our method filters tokens by error type directly in the loss, without additional training stages or a clean validation subset.Self-Cleaning also reweights boundary and type errors separately, but uses a discriminator trained on clean instances.Source: §2 · p. 3

LLMs as Annotators

• LLM annotation studies improve label quality through prompting or hybrid annotation, but do not address robust training on the noisy labels they produce.Examples include GPT-NER, DiZiNER, EvoPrompt, and human–LLM pipelines (Naraki et al., 2024).Source: §2 · p. 3
• Because noisy development data can undermine methods that rely on clean development sets, this work adopts a setting with no clean data.Zhu et al. (2023) find that a small clean set is better spent on fine-tuning than model selection.Source: §2 · p. 3

3.1 Problem Setup

• The setup assumes instance-dependent annotation corruption and trains token-level NER with categorical cross-entropy and a binary reliability mask for clean-test performance.Labels include outside and BIO-tagged entity classes, and the outside class dominates the token distribution.Source: §3.1 · p. 3

3.2 Latent Clean Indicator

• A token’s latent clean indicator is unobserved and is estimated by a MAP decision from the posterior inferred using its maximum predicted-class confidence.The model’s current prediction is the argmax over its class probabilities.Source: §3.2 · p. 3

3.3 Adaptive Threshold Estimation

• A two-component Beta mixture fitted to training confidence scores estimates clean and noisy groups without clean validation labels, with an equal-posterior boundary defining the threshold.Training starts with a fixed threshold, updates it after the first few epochs, and refreshes it twice more at equal intervals.Source: §3.3 · pp. 3–4

3.4 Error-Type-Aware Hard Reweighing

• Error-type-aware reweighting keeps agreement tokens and retains disagreeing tokens only when their error-type-specific model estimates them as clean.The method partitions disagreements into missing-entity and entity errors, with variants splitting entity errors into false positives and type errors; it also compares against one threshold for all misclassified tokens. Tokens from either partition may also involve boundary errors.Source: §3.4 · p. 4

4 Experimental Setup

• Models are trained on noisy annotations, selected using noisy validation data, and evaluated on a clean gold-label test split to isolate annotation-noise effects.The protocol assumes no clean data is available at any stage and considers annotation sources with varied noise levels and error profiles.Source: §4 · p. 4

4.1 Datasets

• The four datasets span label sets of 2 types in BC5CDR, 4 in NoiseBench and Wikigold, and 18 in OntoNotes.Main experiments use LLM-annotated versions; evaluation also includes distant-supervision labels, and NoiseBench has five further noisy variants.Source: §4.1 · p. 4

4.2 LLM Annotation

• LLM annotations come from gpt-oss-120b and Qwen3-235B-A22B-Instruct-2507 using four prompting approaches, with three label variants selected per dataset.The prompting approaches are basic prompting, schema prompting, DiZiNER, and EvoPrompt.Source: §4.2 · p. 5

4.3 Baselines

• Baselines include categorical cross-entropy and four noise-robust losses: GCE, focal loss, BMM bootstrap loss (BMMb), and corrected NLL (NLLc).GCE is cited to Zhang and Sabuncu (2018) and Meng et al. (2021); focal loss to Lin et al. (2018); BMMb to Arazo et al. (2019); and NLLc to Jiang et al. (2021).Source: §4.3 · p. 5

5.1 LLM Annotation Errors

• Error profiles vary by dataset and prompt: BC5CDR errors are mostly missing mentions, while NoiseBench prompts can produce many type errors.In the highest-quality labels for OntoNotes, NoiseBench, and Wikigold, no single error type accounts for more than 40% of erroneous mentions.Source: §5.1 · p. 5, Table 1 · p. 5

5.2 Evaluation of Error-Type-Aware Losses

• Lmissing leads on all OntoNotes and BC5CDR variants, while Lentity leads on NoiseBench and different losses lead across WikiGold variants.Lmissing improves over cross-entropy by approximately 1 F1 point on average; on WikiGold, LFP is best on average, with GPT-OSS EvoPrompt increasing from 60.2 to 64.8. None of the existing noise-robust losses outperforms the proposed variants on any dataset.Source: §5.2 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6
• Lentity is broader and more effective than further splitting its targets into LFP and Ltype, while separating observed-label groups in Lmissing and Lentity beats global Lall.Patterns are similar with XLM-RoBERTa-large; on Wikigold, Lmissing performs best with that model, while Lentity performs best with DistilBERT, though the DistilBERT losses are close.Source: §5.2 · p. 6

5.3 Upper Bounds for Loss Masking

• Ideal Masking approaches clean-training F1 on NoiseBench, while Ideal Lall also improves results under noise but by a smaller margin.Ideal Masking removes all erroneous tokens; Ideal Lall masks erroneous tokens only when the model still misclassifies them. Practical Lall performs poorly across noise types and is often worse than cross-entropy, while Lmissing and Lentity outperform the baseline on different label variants.Source: Figure 2 · p. 7, §5.3 · p. 7

5.4 Masking Dynamics

• Lmissing’s adaptive threshold generally rises in later epochs, while the number of masked tokens starts high and declines, with higher noise producing more masking.The token-count pattern is consistent across noise levels despite differences in threshold trajectories.Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 Why Combined Masking Underperforms

• Lmissing,entity masks over twice as many tokens as either Lmissing or Lentity alone, and its training loss falls while noisy-development F1 fails to improve.This supports the interpretation that simultaneously masking both groups removes too much information for effective learning.Source: Figure 4 · p. 8, §5.5 · p. 8
• Lmissing is the safer default when no error type dominates; use it for missing mentions and Lentity for false positives or type errors.Masking both token groups simultaneously removes too much training signal, whereas addressing each group separately is more effective.Source: §5.5 · pp. 7–8

6 Conclusion

• Variable annotation quality in LLM-generated supervision can substantially affect downstream model performance, particularly for information extraction tasks like NER.Source: §6 · p. 8
• Test F1 scores improve by between 0.8 and 2.0 percentage point per dataset, and targeted error-type-aware masking is more effective than uniform masking across all error types.The evaluation covers four datasets, each with three LLM-generated label variants.Source: §6 · p. 8
• The proposed label noise-robust loss reweighing approach targets different types of NER errors separately.It is evaluated across four datasets and three LLM-generated label variants per dataset.Source: §6 · p. 8
• The method can be applied in a single fine-tuning run without additional resources, making it practical for small domain-specific NER models trained on LLM-annotated data.Source: §6 · p. 8

Limitations

• Noise-robust losses are most effective when aligned with the annotation model’s error distribution, which cannot be known in advance without a small set of clean labels.The authors found no unified error mode across their investigation.Source: Limitations · p. 9
• The need for clean labels to measure the annotation model’s error distribution limits advance selection of the most effective noise-robust loss.Source: Limitations · p. 9
• Combining masking approaches based on a unified error mode degraded performance across the investigated benchmarks.The authors report that they did not find a single unified error mode.Source: Limitations · p. 9
• The method assumes errors can be explained by two Beta components and that the resulting posterior is monotone in score, neither of which is guaranteed.With fewer examples, fewer classes, or different distributions, thresholds may be unreliable and performance may degrade.Source: Limitations · p. 9
• Experiments covered only DistilBERT and XLM-RoBERTa on English data, leaving transfer to autoregressive models and low-resource languages untested.Source: Limitations · p. 9
• Evaluation was limited to token-level NER and did not consider span-level classification, despite its popularity in cross-encoder and bi-encoder generalist models trained on LLM-generated data.Source: Limitations · p. 9

A.1 Datasets

• Wikigold was split 80%/10%/10% by document boundaries and sentence order, while OntoNotes training used 20% of its original set and its full test set.NoiseBench, BC5CDR, and OntoNotes used their original train-test splits.Source: §A.1 · p. 12
• Expert labels came from Flair; distant labels came from BOND for OntoNotes and Wikigold and AutoNER for BC5CDR.The authors selected three representative noisy variants per dataset from 4–6 LLM-generated variants, including the highest-quality label and a range of noise levels.Source: §A.1 · p. 12, §A.2 · p. 12

A.2 LLM Annotation

• Annotation used Basic prompts for all datasets and Schema prompts for Wikigold and NoiseBench, with adapted EvoPrompt and DiZiNER refinement procedures.Refinement used 100 unlabeled training sentences, with two EvoPrompt iterations and four DiZiNER cycles; the annotators were gpt-oss-120b and Qwen3-235B-A22B-Instruct-2507.Source: §A.2 · p. 12

A.3 Models and Fine-Tuning

• Fine-tuning used DistilBERT and XLM-RoBERTa (large) for 10 epochs, with a linear scheduler, 0.1 warmup, 0.01 weight decay, and three random seeds.NoiseBench and Wikigold models received 64 sentences of document context; learning rate and batch size were tuned separately for each dataset and model using noisy development performance.Source: §A.3 · p. 13

A.4 Baseline Losses

• Baseline tuning tried Focal-loss γ values of 0, 1, 2, and 3 and GCE q values of 0.3, 0.5, 0.7, and 0.9.Source: §A.4 · p. 13

A.5 Error-Type-Aware Losses

• Error-type-aware losses used a 0.3 posterior cutoff, a 150-iteration unmasked warmup, and initial confidence thresholds of 0.4 for Lmissing and 0.85 for Lentity.Thresholds were updated after epochs 1, 3, and 6 by fitting a BMM for 10 iterations.Source: §A.5 · p. 13
• An entity-coverage guard applied masking only after the model predicted at least 10% of observed non-O tokens in a batch, preventing collapse when positive examples were absent.The guard was used for Lentity, Lall, LFP, Ltype, and Lmissing,entity.Source: §A.5 · p. 13

B LLM Error Propagation

• Around 80% of missing-mention and partial-match errors are LLM-matched, versus 55% of type errors and 63% of hallucinated errors.The analysis compares direct-prompt test labels with predictions from a model fine-tuned on the corresponding LLM-labeled training set.Source: Table 6 · p. 13, B LLM Error Propagation · p. 13

C Results with XLM-RoBERTa (large)

• The main results tables in this section use XLM-RoBERTa (large), unlike the main paper’s DistilBERT results except for Figure 1.Source: C Results with XLM-RoBERTa (large) · p. 13

D Results on Extended Noisy Variants

• The evaluation covers extended noisy-label variants and a dataset without noise.The section identifies Tables 9 and 10 as the model-specific result tables.Source: D Results on Extended Noisy Variants · p. 13
• Table 9 reports DistilBERT test F1 for error-type-aware losses against cross-entropy and global masking on extended label variants.Source: Table 9 · p. 15
• Table 10 reports XLM-RoBERTa-Large test F1 for error-type-aware losses against cross-entropy and global masking on extended label variants.Source: Table 10 · p. 15