PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossNER: Evaluating Cross-Domain Named Entity Recognition

Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, Pascale Fung

arXiv:2012.04373v2cs.CLcs.AI

TL;DR

Cross-domain NER seeks to adapt models to new domains with few labeled examples, but existing benchmarks often lack focused domains and specialized entity types. The paper introduces CrossNER and domain-related corpora, then evaluates corpus selection and masking strategies for domain-adaptive pre-training. Results show that specialized-entity corpora and more challenging span-level masking improve adaptation, although the task remains difficult in low-resource settings.

  • Problem

    Existing NER benchmarks often lack domain-specialized entity types or focus on a specific domain, weakening cross-domain evaluation.

  • Method

    The paper introduces CrossNER across five diverse domains with specialized categories and studies domain corpus levels and masking strategies for domain-adaptive pre-training.

  • Results

    Specialized-entity corpus emphasis and span-level pre-training consistently improve NER domain adaptation, and the proposed method consistently outperforms existing baselines.

  • Takeaways & Limitations

    CrossNER supplies a focused low-resource benchmark and corpora for studying NER domain adaptation across diverse specialized domains.

  • Takeaways & Limitations

    Domain-adaptive pre-training faces data scarcity in low-resource domains, especially AI, and the best model remains imperfect with limited target-domain samples.

Abstract

from arXiv · show

Cross-domain named entity recognition (NER) models are able to cope with the scarcity issue of NER samples in target domains. However, most of the existing NER benchmarks lack domain-specialized entity types or do not focus on a certain domain, leading to a less effective cross-domain evaluation. To address these obstacles, we introduce a cross-domain NER dataset (CrossNER), a fully-labeled collection of NER data spanning over five diverse domains with specialized entity categories for different domains. Additionally, we also provide a domain-related corpus since using it to continue pre-training language models (domain-adaptive pre-training) is effective for the domain adaptation. We then conduct comprehensive experiments to explore the effectiveness of leveraging different levels of the domain corpus and pre-training strategies to do domain-adaptive pre-training for the cross-domain task. Results show that focusing on the fractional corpus containing domain-specialized entities and utilizing a more challenging pre-training strategy in domain-adaptive pre-training are beneficial for the NER domain adaptation, and our proposed method can consistently outperform existing cross-domain NER baselines. Nevertheless, experiments also illustrate the challenge of this cross-domain NER task. We hope that our dataset and baselines will catalyze research in the NER domain adaptation area. The code and data are available at https://github.com/zliucr/CrossNER.

Introduction

CrossNER addresses weaknesses in existing cross-domain NER evaluation by providing specialized, diverse target domains and analyzing domain-adaptive pre-training strategies. Experiments show benefits from specialized-entity corpora and span-level masking, while the task remains challenging in low-resource settings.

  • Motivation: Existing cross-domain NER benchmarks often use target domains close to the source or broad in topic, with limited specialized entity categories.These limitations reduce their effectiveness for evaluating recognition of entities tied to particular target domains.
  • Dataset: CrossNER is a human-annotated dataset spanning politics, natural science, music, literature, and artificial intelligence, with domain-specific entity categories.Each target domain has approximately 1,000 development and test examples and only 100 or 200 training examples in the low-resource setting.
  • Method: The paper provides five corresponding unlabeled domain-related corpora and studies corpus size and masking strategies for domain-adaptive pre-training.The experiments continue pre-training language models such as BERT before NER adaptation.
  • Results: Emphasizing the partial corpus containing specialized entities consistently improves domain adaptation, while span-level pre-training consistently outperforms original masked language modeling.The span strategy masks contiguous random spans rather than random tokens.
  • Results: The benchmark remains challenging, especially when only a few target-domain training samples are available.

Related Work

Prior cross-domain NER benchmarks commonly use general or heterogeneous domains and entity inventories, while a smaller set of datasets focuses on specific domains. Existing methods address adaptation through label mapping, transfer learning, entity capture, and target-domain knowledge injection.

  • Existing NER Datasets: OntoNotes covers six genres, but they are either relatively close, such as newswire and broadcast news, or broad, such as web data and magazines.
  • Existing NER Datasets: Biomedical NER and CORD-NER are identified as datasets that focus on specific domains.
  • Cross-Domain NER Methods: Cross-domain NER methods have used label-space mapping, double transfer learning, two-stage entity capture, and target-domain knowledge injection.These approaches address adaptation and generalization across source and target domains.

The CrossNER Dataset

CrossNER is a human-annotated dataset spanning five diverse domains with domain-specialized entity categories, supported by unlabeled domain corpora for adaptation research.

  • The authors construct five unlabeled domain-specific corpora from Wikipedia and use them both for annotation sampling and domain-adaptive pre-training.
  • Each sampled sentence is pre-annotated using the DBpedia Ontology before review by two trained annotators and a final NER expert.
  • The AI domain has only a few related Wikipedia pages, creating a scarcity challenge for domain-adaptive pre-training that typically needs many unlabeled sentences.
  • The dataset uses relatively small training sets to test fast adaptation, while hierarchical entity categories require context-sensitive labeling.
  • Vocabulary overlaps between domains are generally small, indicating that the collected domains are diverse.

Domain-Adaptive Pre-training

The paper investigates domain-adaptive pre-training with different corpus granularities and masking strategies, including an entity-focused corpus mixture and span-level masking.

  • The pre-training design addresses large, noisy corpora by testing whether extracting indispensable, entity-rich content can match or improve cross-domain performance.
  • The study compares domain-level, entity-level, and task-level corpora for domain-adaptive pre-training.Entity-level data contains sentences with plentiful entities, whereas task-level data contains domain-specialized entities and is expected to be smaller.
  • The integrated corpus upsamples task-level data before combining it with entity-level data, increasing emphasis on task-relevant sentences.The task-level corpus is doubled in size before integration.
  • The authors replace BERT’s token-level masking with span-level masking by masking contiguous random spans.The remaining masking and replacement procedure follows BERT’s masked language modeling strategy.
  • Span-level masking creates a more challenging prediction task because the model must predict an entire span rather than an individual token.The paper uses predicting “San Francisco” versus “San” as an intuition for this difference.

Experiments

Experiments fine-tune BERT across source and target domains under multiple adaptation settings and compare proposed configurations with established cross-domain NER baselines.

  • BERT is fine-tuned for NER using CoNLL2003 Reuters News data as the source domain and five CrossNER domains as target domains.
  • The experiments apply domain-adaptive pre-training before source or target training and vary corpus level and masking strategy.
  • The adaptation settings are direct target fine-tuning, source pre-training followed by target fine-tuning, and joint source-target training.Target data is upsampled in joint training to balance source and target sample sizes.
  • The comparison includes BiLSTM-CRF, Coach, a language-modeling and NER transfer method, and a BERT-based multi-cell compositional LSTM.
  • Table 2 reports F1-scores for the proposed methods and baseline models, averaged over three runs.

Results & Analysis

Experiments show that entity-focused corpora and span-level masking improve domain-adaptive pre-training, while source-domain supervision further helps cross-domain NER. Performance remains challenging with scarce target data, and gains from larger corpora diminish at high coverage.

  • Corpus selection: Entity-level and task-level corpora match or outperform domain-level corpora despite being substantially smaller.The integrated corpus balances corpus size with content explicitly related to NER in the target domain.
  • Masking strategies: 1.34% F1-score separates span-level from token-level masking on the integrated corpus in Pre-train then Fine-tune.Adding span-level masking with the integrated corpus to Jia and Zhang (2020) further improves average F1-score by 2.16%.
  • Future work: The study leaves exploration of additional masking strategies and DAPT methods for future work.This limitation follows the observed benefits of span-level masking and the integrated corpus.
  • Source-domain supervision: 3.45% average F1-score is gained by Pre-train then Fine-tune without DAPT over Directly Fine-tune, while Jointly Train gains 3.08%.The best model outperforms the existing state-of-the-art model in all five domains, but its average F1-score remains below 70%.
  • Unlabeled corpus size: Larger unlabeled corpora generally improve DAPT performance, although gains become comparatively smaller at 75% corpus coverage or higher.The analysis uses different percentages of the music domain’s integrated corpus and compares masking strategies across three settings.
  • Unlabeled corpus size: 1% of the music integrated corpus, or 1.48M, yields little improvement for either masking strategy; span-level masking gains an advantage as corpus size increases.The authors hypothesize that span-level masking learns domain text more efficiently, whereas token-level masking requires a larger corpus.
  • Few-shot adaptation: At 10 target-domain samples, performance becomes extremely difficult, while span-level pre-training generally remains stronger than token-level pre-training.When the sample size is 40 or lower, Pre-train then Fine-tune exceeds Directly Fine-tune by approximately 10% F1.
  • Fine-grained categories: Source-domain NER data can hurt specialized categories such as artist and band, whereas DAPT greatly improves categories including song, band, and album.In the best model, 84.81% of person entities are misclassified as artist, reflecting confusion from the hierarchical category structure.

Conclusion

The paper introduces CrossNER and domain-related corpora for cross-domain NER, evaluates adaptation strategies, and reports consistent baseline improvements. However, the best model remains imperfect when target-domain training data are limited.

  • CrossNER spans five diverse domains with specialized entity categories, and accompanying domain-related corpora support domain-adaptive pre-training studies.
  • The paper reports benchmark results for existing NER models and analyzes domain-corpus size and pre-training strategies for cross-domain adaptation.
  • The proposed method consistently outperforms existing baselines across the cross-domain NER experiments.
  • The best model is not yet perfect, especially when the number of target-domain training samples is limited.

The CrossNER Dataset

CrossNER uses trained annotators, multiple review stages, and domain-specific instructions to label entities across five specialized domains. Its rules resolve nested entities and prioritize specialized labels over broader categories.

  • Annotator Training: Annotators are trained with instructions and 100 expert-annotated examples before domain annotation begins.Experts inspect checking results and explain annotator mistakes during training.
  • Annotation Instructions: The annotation guidance separates general rules from domain-specific instructions.
  • General Instructions: Each sample receives annotations from two trained annotators and final adjudication by an NER expert.The process includes independent annotation, second-stage checking, and expert resolution of disagreements.
  • General Instructions: Nested entities are resolved by labeling the larger span, while hyperlinks serve as likely entity cues during annotation.For example, “Fellow of the Royal Society” receives the award label instead of the nested organization span.
  • Domain-Specific Instructions: Politics uses categories including politician, political party, election, event, country, and miscellaneous alongside general entity types.Election events receive the election label, and political parties receive the political-party label rather than broader alternatives.
  • Domain-Specific Instructions: Natural science covers biology, chemistry, and astrophysics with categories such as scientist, discipline, enzyme, protein, chemical compound, theory, and award.Scientists replace person labels, enzymes replace protein labels, and the source is Wikipedia rather than academic papers.
  • Domain-Specific Instructions: Music and literature define specialized categories including genres, songs, bands, albums, musical artists, books, writers, poems, magazines, and literary genres.Domain roles override broader labels, such as band over organization, musical artist over person, and writer over person.
  • Domain-Specific Instructions: Artificial intelligence includes field, task, product, algorithm, researcher, metrics, programming language, and conference categories alongside general types.Researchers receive researcher labels instead of person labels, while fields include machine learning and tasks include machine translation.

Training Details

Training uses domain-adaptive pre-training before source or target training, compares corpus overlap and masking choices, and evaluates models with F1-score. The setup includes three adaptation settings and tuned optimization parameters.

  • BERT undergoes domain-adaptive pre-training before source- or target-domain training when unlabeled domain-related corpora are used.
  • Models add a linear layer to BERT, fine-tune the whole model for NER, and use F1-score for evaluation.
  • The training configuration searches batch sizes 16 and 32 with learning rates 1 × 10^-5, 3 × 10^-5, and 5 × 10^-5.
  • Vocabulary-overlap analysis compares unlabeled domain-related corpora across Reuters News, natural science, literature, and artificial intelligence.

Supplementary Experiments

Supplementary experiments test whether NER-model-generated corpora can replace DBpedia Ontology-based corpus construction. The resulting DAPT achieves comparable results, indicating that corpus construction does not rely on DBpedia Ontology.

  • NER-model-based construction replaces DBpedia Ontology by recognizing and categorizing entities in domain-related corpora.The process uses source and target training data, then extracts entity-rich sentences and selects sentences containing domain-specialized entities.
  • DAPT with NER-model-based corpora achieves comparable results to DAPT with DBpedia Ontology-based corpora.The experiments show that constructing different corpus levels does not rely on DBpedia Ontology.
  • The supplementary comparison evaluates proposed methods across five domains and three settings using span-level masking.
Loading 2012.04373v2…