PaperlayerView exact revision on arXiv

Source-linked AI summary

Type-Balanced Contextual Learning for Incremental Named Entity Recognition

Duzhen Zhang, Yahan Yu, Xiuyi Chen, Chenxing Li, Dong Yu

arXiv:2608.31038v1cs.CL

TL;DR

INER must continually learn new entity types while avoiding catastrophic forgetting and non-entity semantic shift, yet pseudo-labeling introduces biased contexts around old entities. TBCL uses sentence-duplet learning and contextual consistency loss to correct these contexts, and experiments across ten settings and three datasets show superiority over prior INER methods.

  • Problem

    INER faces catastrophic forgetting, non-entity semantic shift, and biased contexts in pseudo-labeling-based methods, which are associated with degraded old knowledge and new-knowledge overfitting.

  • Method

    TBCL combines sentence-duplet learning with contextual consistency loss to balance contextual information between old and new entity types.

  • Results

    TBCL surpasses previous state-of-the-art INER approaches across ten settings on CoNLL2003, I2B2, and OntoNotes5.

  • Takeaways & Limitations

    TBCL provides a context-analysis approach for reducing forgetting of old entity types and overfitting to new ones in pseudo-labeling-based INER.

Abstract

from arXiv · show

Incremental Named Entity Recognition (INER) stands as a pivotal task in information extraction, emphasizing the successive identification of new entity types within unstructured text. Faced with the continuous influx of entity types, INER grapples with two significant challenges: the widespread issue of catastrophic forgetting and the unique shift issue of the non-entity type semantics. While pseudo-labeling-based INER methods have proven effective in addressing these challenges, a previously overlooked issue arises: the biased context problem. Our analysis shows that, in new sentences, the contextual associations of tokens representing old entity types exhibit a significantly stronger bias towards new entity types compared to their contexts in old sentences. This tendency intensifies the degradation of old knowledge while promoting the overfitting of new knowledge. To solve this biased context, we propose a Type-Balanced Contextual Learning (TBCL) method, featuring a sentence-duplet learning scheme and a contextual consistency loss. This approach offers a fresh perspective for INER through context analysis. Extensive experiments across ten INER settings on three highly recognized datasets showcase the efficacy of our TBCL method, highlighting its proficiency in resolving the biased context issue inherent in pseudo-labeling based INER approaches.

I. INTRODUCTION

INER must learn emerging entity types while retaining prior knowledge, but faces catastrophic forgetting, non-entity semantic shift, and biased contexts introduced by pseudo-labeling. TBCL addresses the biased-context problem with sentence-duplet learning and contextual consistency, achieving new state-of-the-art performance across evaluated settings.

  • INER extends named entity recognition to sequentially identify emerging entity types in unstructured text.
  • Catastrophic forgetting causes models to lose previously learned entity knowledge when adapting to new entity types.
  • Semantic shift occurs because previous and future entity types are labeled as non-entity tokens during the current learning step.
  • Pseudo-labeling augments current labels with previous-model predictions to address the non-entity semantic shift.
  • New sentences disproportionately associate old entity tokens with new entity types, intensifying old-knowledge forgetting and new-knowledge overfitting.
  • TBCL combines sentence-duplet learning with contextual consistency loss to correct biased contexts between old and new entity types.
  • Experiments across ten INER settings and three datasets show that TBCL surpasses existing methods and achieves new SOTA performance.

II. RELATED WORK

Related work addresses incremental learning and INER through knowledge distillation, rehearsal, causal modeling, and pseudo-labeling. These methods reduce forgetting or semantic shift, but pseudo-labeling-based RDP leaves a biased-context issue that TBCL targets.

  • Incremental Learning: Incremental learning methods use regularization, dynamic architectures, or replay to preserve earlier knowledge while learning new tasks.
  • Named Entity Recognition: Traditional NER classifies tokens into predefined entity types or the non-entity type, whereas practical systems must accommodate emerging types.
  • INER: Existing INER methods apply knowledge distillation, rehearsal, or causal modeling to reduce catastrophic forgetting.
  • INER: These earlier INER approaches remain limited in handling the non-entity type's semantic shift.
  • Pseudo-labeling-Based INER: RDP combines knowledge distillation with prototypical pseudo-labeling to mitigate catastrophic forgetting and non-entity semantic shift.
  • Pseudo-labeling-Based INER: RDP introduces biased contexts because new sentences predominantly contain new-entity-related contexts around old-entity tokens.
  • Pseudo-labeling-Based INER: TBCL addresses this defect by combining sentence-duplet learning and contextual consistency loss for more balanced learning contexts.

A. Problem Formulation

The formulation trains models sequentially on step-specific datasets whose labels expose both catastrophic forgetting and non-entity semantic shift. The paper then frames RDP as a knowledge-distillation and pseudo-labeling baseline for addressing these issues.

  • A. Problem Formulation: INER trains models sequentially across steps t = 1, 2, ..., T using datasets D1, D2, ..., DT, creating potential forgetting of earlier entity types.
  • A. Problem Formulation: At step t, each dataset contains token sequences Xt and one-hot labels Yt for only the current entity types Et.
  • A. Problem Formulation: Prior and future entity types are merged into the non-entity class eo, producing the semantic-shift problem; entity types remain mutually exclusive across steps.
  • A. Problem Formulation: The current model Mt uses an encoder Ft and linear softmax classifier Ct to recognize all entity types encountered through step t.
  • A. Problem Formulation: The label space is Ct = {eo}∪E1:t, and predictions are represented by the probability matrix Ŷt = Mt(Xt; Θt).
  • A. Problem Formulation: The current-step objective includes cross-entropy over the ground-truth labels, with Θt denoting the trainable parameters of Mt.
  • B. Pseudo-labeling Based INER: RDP combines knowledge distillation with prototypical pseudo-labeling to preserve prior knowledge and create higher-quality pseudo labels.
  • B. Pseudo-labeling Based INER: Knowledge distillation transfers predicted probabilities from Mt−1 to Mt, using only the old label space Ct−1 for the distillation loss.

2) Prototypical Pseudo-label Strategy:

RDP’s prototypical pseudo-label strategy uses old-model predictions and prototype distances to assign more reliable labels to non-entity tokens while preserving current ground-truth labels.

  • Prototypical Pseudo-label Strategy: RDP constructs pseudo-label targets by retaining ground-truth labels for current entity tokens and using old-model predictions over the old label space for non-entity tokens.The target matrix assigns each token to a class, with non-entity tokens receiving pseudo-labels from the previous model.
  • Prototypical Pseudo-label Strategy: The method is designed to reduce errors from old-model predictions and address the semantic shift of the non-entity type.Non-entity tokens can represent previously learned or future entity types, so their old-model predictions provide additional target information.
  • Prototypical Pseudo-label Strategy: Prototype-based weighting adjusts old-model probabilities according to distances between token embeddings and type-specific prototypes.The prototypes are computed as average embeddings of tokens assigned to each entity type, while embedding-prototype distances identify potential outliers.
  • Prototypical Pseudo-label Strategy: The pseudo-labeling cross-entropy loss trains the current model with constructed pseudo labels, while knowledge distillation retains information from the old model.RDP combines these terms as Lrdp = Lpce + αLkd, where α balances their importance.

C. TBCL

RDP mitigates catastrophic forgetting and non-entity semantic shift, but its pseudo-labeled new sentences create biased contexts around old entity types. TBCL is introduced to reduce this bias.

  • C. TBCL: RDP effectively mitigates catastrophic forgetting and semantic-shift problems, yet its updated model encounters biased context between old and new entity types.The new sentences predominantly provide new-entity-related context for tokens belonging to old entity types.
  • C. TBCL: TBCL targets this bias because it can exacerbate forgetting of old entity types while promoting overfitting to new entity types.The method is presented as a way to reduce biased context correlation in the pseudo-labeling-based RDP approach.

1) Sentence-duplet Learning Scheme:

TBCL forms sentence pairs from each new sentence and an erased counterpart, then trains on both using pseudo-labeling, distillation, and contextual consistency objectives.

  • 1) Sentence-duplet Learning Scheme: For each original sentence containing new-entity tokens, TBCL creates a modified sentence by removing those tokens to reduce biased context around old entity types.The original and erased sentences form the sentence-duplet used at incremental step t.
  • 1) Sentence-duplet Learning Scheme: The sentence-duplet learning scheme updates the new model using both the original and erased sentences.The corresponding losses use the supervision signals associated with each member of the pair.
  • 1) Sentence-duplet Learning Scheme: TBCL’s sentence-duplet objective incorporates pseudo-labeling cross-entropy and knowledge distillation through the RDP loss applied to the paired inputs.The figure identifies Lpce, Lkd, and Lctxc as the losses used during subsequent incremental steps.
  • 2) Contextual Consistency Loss: The contextual consistency loss constrains predictions for old-entity-type tokens to remain consistent across original and erased sentence contexts.It compares current-model predictions at positions containing pseudo-labeled old-entity tokens, where new-entity context is present versus removed.
  • 2) Contextual Consistency Loss: TBCL combines the sentence-duplet loss with the contextual consistency loss, weighted by hyperparameter β.The total objective is Ltbcl = Ls-dup + βLctxc.

IV. EXPERIMENTAL SETUP

The experiments follow prior INER setups and evaluate TBCL on three established NER datasets partitioned into incremental slices using greedy sampling.

  • IV. EXPERIMENTAL SETUP: The evaluation matches CFNER and RDP in benchmark datasets, INER settings, competing baselines, metrics, and foundational implementation details.This setup is intended to ensure fair comparison with previous state-of-the-art INER methods.
  • A. Benchmark Datasets: The study uses CoNLL2003, I2B2, and OntoNotes5 as its three NER datasets.Table I summarizes the dataset statistics.
  • A. Benchmark Datasets: Greedy sampling divides each training set into incremental slices with samples from each entity type concentrated in its corresponding slice.Within each slice, labels for learned entity types are retained and all other labels are treated as non-entity types.

B. INER Settings

The study evaluates INER across two incremental scenarios per dataset, comparing TBCL with established and adapted baselines using step-wise Macro-F1 and Micro-F1 metrics.

  • INER scenarios: Each dataset uses two INER scenarios: equal-sized initial and incremental steps, or an initial model trained on half of all entity types.The equal-step scenario is more challenging, while the half-type scenario better reflects models acquiring sufficient knowledge before incremental learning.
  • Baselines: The evaluated baselines include ExtendNER, CFNER, RDP, Only Finetuning, PODNet, LUCIR, and Self-Training.These comprise SOTA INER methods, a lower-bound finetuning method, and computer-vision incremental-learning methods adapted to INER.
  • Baselines: RDP combines task relation distillation with prototypical pseudo-labeling to address catastrophic forgetting and non-entity semantic shift.Its prototype-based strategy uses type-wise embedding distances to reduce pseudo-label noise and refine previous-model predictions.
  • Assessment metrics: Performance is assessed with old-, new-, and all-type Macro-F1, plus all-type Micro-F1, averaged across incremental steps.Step-wise score plots and paired t-tests at significance level 0.05 complement the average metrics.

E. Implementation Details

The experiments use BIO-tagged NER with a BERT-based encoder and investigate quantitative, component, qualitative, and stability questions across incremental settings.

  • Implementation: The NER model uses bert-base-cased as encoder, a fully connected classification layer, and PyTorch with Huggingface’s BERT implementation.Training uses 20 epochs when PG=2 and 10 otherwise, with batch size 8, learning rate 4e−4, and γ=0.01.
  • Research questions: The evaluation asks whether TBCL improves quantitative performance, how its components affect results, and how it compares qualitatively with competitive baselines.These correspond to RQ1, RQ2, and RQ3.
  • Research questions: A fourth question tests stability under different entity-type orders, larger encoder backbones, and finer-grained entity types.This extends evaluation beyond the primary benchmark settings.

A. Main Results (RQ1)

Across ten settings on CoNLL2003, I2B2, and OntoNotes5, TBCL improves over RDP on old, new, and all entity types, with especially large gains in harder settings.

  • I2B2: TBCL improves I2B2 all-type Micro-F1 by 0.75, 0.74, 6.49, and 2.29 across the four reported settings.The corresponding all-type Macro-F1 gains are 4.66, 0.98, 4.28, and 2.79.
  • OntoNotes5: TBCL improves OntoNotes5 all-type Micro-F1 by 1.76, 2.02, 1.52, and 0.50 across the four reported settings.The corresponding all-type Macro-F1 gains are 2.07, 2.11, 3.31, and 1.51.
  • Cross-dataset pattern: The largest I2B2 Micro-F1 gains occur in FG-8-PG-1 (+6.49) and FG-8-PG-2 (+2.29), while OntoNotes5 peaks at +2.02 Micro-F1 and +3.31 Macro-F1.The reported analysis associates larger gains with more challenging multientity-type settings and still finds consistent improvements in simpler settings.
  • Step-wise results: TBCL consistently outperforms RDP in most step-wise evaluations and is reported to mitigate old-type forgetting and new-type overfitting by rectifying biased context.The step-wise comparison is shown for the FG-8-PG-1 I2B2 setting.

C. Ablation Study (RQ2)

Ablations show that both sentence-duplet learning and contextual consistency contribute to TBCL’s INER performance, while qualitative examples compare predictions across multiple entity types.

  • Sentence-duplet learning: Baseline+duplet outperforms Baseline+double, showing that adding erased sentence variants is more effective than merely duplicating original samples.The contrast isolates sentence-pair content from increased sample count.
  • Contextual consistency loss: Adding contextual consistency loss to Baseline+duplet further enhances INER performance by enforcing consistency across contexts.This full variant is denoted Baseline+duplet+ctxc.
  • Qualitative evaluation: The qualitative evaluation uses last-task predictions from FG-8-PG-2 on OntoNotes5, comparing TBCL with CFNER and RDP.The examples use BIO labels for multiple entity types, including Person, Time, Organization, and Product.
  • Qualitative evaluation: The visual examples are presented as evidence that TBCL can sequentially learn and preserve multiple entity types.The comparison includes token-level predictions from CFNER and RDP alongside the input sentence.

E. More Explorations (RQ4)

Additional experiments show that TBCL remains effective with larger encoder backbones, across fine-grained entity types, and under random entity-type orderings. These evaluations support its performance and stability relative to prior methods.

  • Stability Concerning Entity Type Orders: TBCL performs better and more stably than RDP across 10 random entity-type orderings in the I2B2 FG-8-PG-2 setting.Its boxplots show higher Macro-F1 values, a more concentrated distribution, and a smaller variation range for old, new, and all entity types.
  • Different Encoder Backbones: Both RDP and TBCL benefit from larger encoder backbones, while TBCL maintains its performance advantage over strong baselines.On I2B2 under FG-8-PG-2, RDP’s Micro-F1 rises from 80.08 to 82.50 with a larger backbone.
  • Fine-Grained Entity Types: On FEW-NERD, TBCL consistently outperforms all baselines across the 11 incremental steps.The dataset contains 66 fine-grained entity types, and TBCL achieves the highest Macro-F1 at almost all steps and the best average Macro-F1.
  • Overall Evaluation: Across ten INER settings on CoNLL2003, I2B2, and OntoNotes5, TBCL surpasses previous state-of-the-art INER approaches.The method combines sentence-duplet learning with contextual consistency loss to address biased context correlations.
Loading 2608.31038v1…