PaperlayerView exact revision on arXiv

Source-linked AI summary

CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning

Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J. Passonneau, Rui Zhang

arXiv:2109.07589v2cs.CL

TL;DR

Few-shot NER needs representations that generalize beyond source-domain entity classes when labeled target data are scarce. CONTaiNER uses contrastive learning over Gaussian embeddings to optimize inter-token distribution distances, and evaluations report consistent gains across datasets and challenging domain transfers. Its performance remains below supervised NER and is not yet ready for high-stakes deployment.

  • Problem

    Prior few-shot NER approaches learn source-class-specific representations, limiting generalization to unseen target classes and domains.

  • Method

    CONTaiNER uses contrastive learning over Gaussian embeddings to optimize distributional divergence between token entity representations and adapt with few-shot support data.

  • Results

    CONTaiNER consistently outperforms prior state-of-the-art methods across multiple few-shot NER datasets and challenging scenarios.

  • Takeaways & Limitations

    A generalized distribution-optimization objective can support adaptation to new target domains with few support samples.

  • Takeaways & Limitations

    Performance remains low compared with supervised NER using full training data, so the method is not ready for high-stakes deployment.

Abstract

from arXiv · show

Named Entity Recognition (NER) in Few-Shot setting is imperative for entity tagging in low resource domains. Existing approaches only learn class-specific semantic features and intermediate representations from source domains. This affects generalizability to unseen target domains, resulting in suboptimal performances. To this end, we present CONTaiNER, a novel contrastive learning technique that optimizes the inter-token distribution distance for Few-Shot NER. Instead of optimizing class-specific attributes, CONTaiNER optimizes a generalized objective of differentiating between token categories based on their Gaussian-distributed embeddings. This effectively alleviates overfitting issues originating from training domains. Our experiments in several traditional test domains (OntoNotes, CoNLL'03, WNUT '17, GUM) and a new large scale Few-Shot NER dataset (Few-NERD) demonstrate that on average, CONTaiNER outperforms previous methods by 3%-13% absolute F1 points while showing consistent performance trends, even in challenging scenarios where previous approaches could not achieve appreciable performance.

1 Introduction

Few-shot NER must generalize from source-domain supervision to unseen entity classes and domains despite limited labeled data. CONTaiNER addresses this with contrastive learning over Gaussian embeddings and reports broad gains across evaluation settings.

  • Few-shot NER is needed because supervised NER models require large-scale human-annotated datasets.
  • Prior few-shot NER methods can tie representations to source classes, causing useful features for target entities to be lost.
  • CONTaiNER decreases distances between similar-entity token embeddings and increases distances between dissimilar ones.This contrastive objective is intended to capture label dependencies and address problems with O-tokens.
  • CONTaiNER optimizes distributional divergence over Gaussian embeddings rather than similarity between point embeddings.Gaussian embeddings model class distributions and support generalized representations for few-sample adaptation.
  • 13% absolute F1-points is the maximum reported average improvement over previous state-of-the-art approaches across diverse tests.Experiments include in-domain and out-of-domain settings on CoNLL ’03, OntoNotes 5.0, WNUT ’17, and I2B2.
  • CONTaiNER outperforms all other state-of-the-art approaches on Few-NERD and establishes a new leaderboard benchmark result.

2 Task Formulation

Few-shot NER trains on one tag set and adapts to a data-scarce target domain with unseen tags and limited support examples. The formulation also specifies IO tagging and episode-based evaluation conventions.

  • Few-shot NER trains in a source domain with tag set {Cs} and tests in a target domain with tag set {Cd}.
  • When source and target tag sets are disjoint, generalizing to unseen test tags is challenging.
  • In an N-way K-shot setting, the target domain contains N tags, each associated with a support set of K examples.
  • The IO tagging scheme marks entity-internal tokens with I and all other tokens with O.
  • Episode evaluation computes micro-F1 across test episodes containing K-shot support sets and K-shot unlabeled query sets.

3 Method

CONTAINER learns generalized token representations by contrasting Gaussian embeddings, then adapts them to target domains with few support samples and nearest-neighbor inference.

  • 3 Method: CONTAINER optimizes distributional divergence between token entity representations rather than focusing on label-specific attributes.Gaussian embeddings model entity class distributions and support generalized token representations for few-sample target-domain adaptation.
  • 3.1 Model: BERT contextual token representations are projected into Gaussian embeddings using separate networks for their mean and diagonal covariance.The model uses fµ and fΣ to produce the Gaussian distribution parameters from token representations.
  • 3.2 Training in Source Domain: The contrastive objective treats same-label tokens as positive pairs and compares their Gaussian embeddings using bidirectional KL-divergence.KL-divergence is computed between valid token pairs sampled within each training batch.
  • 3.3 Finetuning to Target Domain using Support Set: After source-domain training, CONTAINER fine-tunes on a single batch of target support samples and stops when the contrastive loss no longer decreases.The procedure uses early stopping because few-shot target domains lack held-out validation data.
  • 3.4 Instance Level Nearest Neighbor Inference: For inference, the projection heads are discarded and test tokens receive the label of the nearest support token in PLM representation space.The authors report that representations before projection contain more information for inference.

4 Experiment Setups

The experiments evaluate CONTAINER across traditional tag-set and domain-transfer settings, Few-NERD splits, and multiple domains using established few-shot NER baselines. The setups distinguish fine-grained class separation from domain transfer and include challenging cross-domain and coarse-type-disjoint scenarios.

  • 4 Experiment Setups: The evaluation spans General, Medical, News, Social, GUM, and Few-NERD datasets across multiple few-shot NER settings.GUM includes interviews, news articles, instrumental texts, and travel guides, making it particularly varied.
  • 4 Experiment Setups: CONTAINER is evaluated in tag-set extension and domain-transfer tables alongside Few-NERD INTRA and INTER evaluations.The cited table captions identify the F1-score evaluations and their respective settings.
  • 4 Experiment Setups: The baselines include ProtoBERT, NNShot, and Structshot, covering prototypical-network, nearest-neighbor, and Viterbi-decoding approaches.The comparisons use traditional NER datasets and the Few-NERD leaderboard setting.
  • 4.1 Tag-set Extension Setting: Tag-set extension splits OntoNotes entity classes into three disjoint groups, training on two groups and adapting to the remaining group with few-shot support.The setup represents new entity types appearing within an existing text domain.
  • 4.2 Domain Transfer Setting: Domain transfer trains on OntoNotes and evaluates few-shot performance on I2B2, CoNLL, WNUT, and GUM target domains.CoNLL target entities are fully contained within OntoNotes entities, while the other domains have almost no intersection with OntoNotes.
  • 4.3 Few-NERD Setting: Few-NERD (INTRA) separates train, development, and test fine-grained types by coarse-grained categories, whereas Few-NERD (INTER) shares coarse types but keeps fine-grained types disjoint.The INTRA split is described as more challenging because coarse-grained types cannot be shared across partitions.

5 Results and Analysis

CONTAINER consistently outperforms baseline approaches across few-shot NER scenarios, with especially strong behavior in challenging domain and label-transfer settings. Analyses indicate that distribution modeling, fine-tuning, and conditional decoding each contribute under different conditions.

  • Overall Results: CONTAINER outperforms all baseline approaches overall and remains effective in challenging scenarios where other methods perform poorly.FEW-NERD intra involves non-overlapping coarse-grained train and test entity types, substantially harming baselines while CONTAINER still performs well.
  • Overall Results: CONTAINER consistently outperforms state-of-the-art models on challenging unseen-domain transfer, with particularly prominent gains in 5-shot cases.On GUM, baseline models perform poorly, whereas CONTAINER maintains a significant margin; StructShot is marginally higher in a few 1-shot cases.
  • Representation Analysis: Distribution modeling through contrastive Gaussian Embedding optimization helps CONTAINER make better use of multiple few-shot samples.The analysis links this behavior to distribution modeling rather than to a single class-specific representation.
  • Effect of Model Fine-tuning: Fine-tuning with support sets improves few-shot performance, with a more marked effect in 5-shot prediction.The case study evaluates OntoNotes tag-extension using PERSON, DATE, MONEY, LOC, FAC, and PRODUCT entities.
  • Modeling Label Dependencies: Viterbi decoding improves domain-transfer results but provides almost no improvement for tag-set extension and FEW-NERD.The paper attributes this difference to label-dependency information supplied by decoding when train and test text domains do not overlap.

6 Related Works

Related work applies meta-learning, contrastive learning, Gaussian embeddings, and supervised or metric-based methods to few-shot learning and Few-Shot NER. Prior Few-Shot NER systems include prototypical, nearest-neighbor, Viterbi-decoding, and noisy-supervised-pretraining approaches.

  • Meta Learning: Prototypical Networks learn class prototypes and assign test samples according to the nearest prototype, with applications beyond computer vision.The approach has also been used in NLP and Few-Shot NER.
  • Contrastive Learning: Contrastive learning develops representations by contrasting positive and negative samples, including modern in-batch negative-sampling approaches.SimCLR uses image augmentation, while supervised contrastive learning adapts the framework to improve supervised representation learning.
  • Gaussian Embedding: Gaussian embeddings represent words or graph entities as distributions, enabling uncertainty representation through variances.Prior work also reported strong performance and improved generalization with Gaussian embedding methods.
  • Few-Shot NER: Few-Shot NER methods include prototypical networks, supervised feature extractors with Viterbi decoding, and noisy supervised pretraining.Noisy-pretraining methods require access to a large-scale noisy NER dataset such as WiNER.

7 Conclusion

The paper concludes that CONTAINER combines contrastive learning with Gaussian embeddings to optimize inter-token distribution distance for Few-Shot NER. Its generalized objective supports adaptation across multiple datasets and challenging scenarios, while broader applications remain an open direction.

  • Conclusion: CONTAINER models Gaussian embeddings and optimizes inter-token distribution distance to learn a class-agnostic feature extractor.The generalized objective is intended to avoid pitfalls of prior Few-Shot NER methods.
  • Conclusion: CONTAINER consistently outperforms prior state-of-the-art methods across traditional and recent Few-Shot NER datasets, including challenging scenarios.The conclusion also reports that the model can use few-sample support data to adapt to new target domains.
  • Conclusion: The efficacy of distribution-optimization-based contrastive learning in other domains remains an open research direction.The paper identifies broader-domain investigation as a future opportunity.

Ethics Statement

CONTAINER achieves state-of-the-art Few-Shot NER performance using Gaussian-embedding contrastive learning, but its overall performance remains below fully supervised NER. The authors therefore caution against deployment in high-stakes domains and identify substantial room for improvement.

  • Ethics Statement: CONTAINER achieves state-of-the-art Few-Shot NER performance by leveraging Gaussian-embedding-based contrastive learning.
  • Ethics Statement: CONTAINER remains below supervised NER that uses the full training dataset and is not ready for deployment in high-stakes domains.The paper gives the medical domain and I2B2 dataset as examples.

A Implementation Details

The experiments use fixed comparison settings and examine how finetuning objectives behave with different numbers of support examples. Gaussian KL-divergence is advantageous in 5-shot adaptation, whereas Euclidean mean-embedding finetuning is better in 1-shot domain transfer.

  • Implementation settings: CONTAINER uses the same hyperparameters as Yang and Katiyar (2020), fixes Gaussian Embedding dimension to l = 128, and matches the backbone encoder across methods.bert-base-cased is used for Tag-Set Extension and Domain Transfer, while bert-base-uncased is used for Few-NERD.
  • Finetuning objectives: With no prior target-class information, a 1-shot example may not adequately estimate a class distribution's variance.This motivates optimizing the mean embedding with Euclidean distance for 1-shot adaptation.
  • Representation analysis: Figure 3 compares support-set and test-set representations before and after finetuning in an OntoNotes tag-extension task.The visualization is used to examine target-domain adaptation from few-shot support examples.
  • Finetuning objectives: KL-divergence finetuning gives superior results in 5-shot WNUT domain transfer, while Euclidean mean-embedding finetuning performs better in 1-shot adaptation.The domain-transfer model is trained on out-of-domain data using KL-Gaussian representations.

D Comparison of Different Training Objectives

The comparison evaluates Gaussian Embedding with KL-divergence against point embeddings using Euclidean distance or cosine similarity. Gaussian Embedding with KL-divergence largely outperforms point embedding across the tested distance metrics, while the visualizations show more separated unseen-class clusters before finetuning.

  • Representation comparison: Before finetuning, point embeddings with Euclidean distance produce intermixed class representations, whereas Gaussian Embeddings with KL-divergence generate clusters for different unseen classes.Figure 3 presents t-SNE visualizations of support and test representations in an OntoNotes tag-extension task.
  • Quantitative comparison: Gaussian Embedding with KL-divergence largely outperforms point embedding irrespective of the distance metric used.Table 8 compares point embeddings with Euclidean distance and cosine similarity against Gaussian Embedding with KL-divergence on OntoNotes tag extension.

E Embedding Quality: Before vs. After Projection

The study compares representations before and after CONTAINER's Gaussian projection layer. Representations before projection achieve higher performance because the post-projection layer is directly adjacent to the contrastive objective and loses information.

  • Performance comparison: Representations before the Gaussian projection layer achieve higher F1 scores than representations after projection.Table 9 reports this comparison on OntoNotes Group A.
  • Interpretation: The post-projection representation loses information because it is directly adjacent to the contrastive objective.This information loss explains why the pre-projection representation performs better.

F NER Prediction Examples

Qualitative OntoNotes examples compare CONTAINER and StructShot on six target entity types. CONTAINER produces fewer misclassifications and better entity detection, while StructShot often confuses entity and non-entity tokens.

  • Qualitative errors: StructShot often fails to distinguish non-entity from entity tokens and misclassifies non-entity tokens as target classes.These errors appear in the qualitative OntoNotes prediction examples.
  • Qualitative comparison: CONTAINER shows lower misclassifications and better entity detection than StructShot in the reported examples.The passage contrasts the models qualitatively rather than providing a numerical error rate.
  • Evaluation setup: Table 10 compares CONTAINER and StructShot when PERSON, DATE, MONEY, LOC, FAC, and PRODUCT are target few-shot entities.The models are trained on the remaining OntoNotes entity types.
Loading 2109.07589v2…