PaperlayerView exact revision on arXiv

Source-linked AI summary

Template-Based Named Entity Recognition Using BART

Leyang Cui, Yu Wu, Jian Liu, Sen Yang, Yue Zhang

arXiv:2106.01760v1cs.CL

TL;DR

Few-shot NER must transfer knowledge from resource-rich domains to target domains with different label sets, while similarity-based methods do not fully exploit NER model parameters. The paper addresses this with template-based BART, treating span classification as sequence-to-sequence language-model ranking. It reports competitive resource-rich performance and significant improvements over traditional methods in cross-domain and few-shot benchmarks.

  • Problem

    Few-shot NER targets domains with scarce labels and different label sets, while existing distance-based methods do not update NER model parameters for representation transfer.

  • Method

    The method fine-tunes BART on original sentences paired with templates filled by candidate spans and entity types, then ranks templates to classify each span.

  • Results

    The model achieves competitive CoNLL03 performance and significantly outperforms traditional sequence-labeling and distance-based methods on cross-domain and few-shot NER benchmarks.

  • Takeaways & Limitations

    The template-based approach can handle arbitrary new entity categories without changing the output layer and supports continual learning.

  • Takeaways & Limitations

    Performance is limited when label semantics differ substantially between high-resource and low-resource domains, because the method relies on label semantics.

Abstract

from arXiv · show

There is a recent interest in investigating few-shot NER, where the low-resource target domain has different label sets compared with a resource-rich source domain. Existing methods use a similarity-based metric. However, they cannot make full use of knowledge transfer in NER model parameters. To address the issue, we propose a template-based method for NER, treating NER as a language model ranking problem in a sequence-to-sequence framework, where original sentences and statement templates filled by candidate named entity span are regarded as the source sequence and the target sequence, respectively. For inference, the model is required to classify each candidate span based on the corresponding template scores. Our experiments demonstrate that the proposed method achieves 92.55% F1 score on the CoNLL03 (rich-resource task), and significantly better than fine-tuning BERT 10.88%, 15.34%, and 11.73% F1 score on the MIT Movie, the MIT Restaurant, and the ATIS (low-resource task), respectively.

1 Introduction

Few-shot NER must transfer knowledge across domains with different entity categories, but existing distance-based methods do not update model representations. The paper proposes template-based BART, which classifies candidate spans through template scores and supports new categories without changing the output layer.

  • NER identifies mention spans according to predefined categories such as location, person, and organization.
  • Few-shot domain adaptation is difficult because target domains may use entity categories different from those in the resource-rich source domain.Traditional softmax and CRF outputs require consistent label sets across training and testing.
  • Distance-based methods reduce adaptation cost but do not update NER model parameters and assume similar writing patterns across domains.Their in-domain performance is relatively weak, and differing target-domain writing styles can hinder performance.
  • Template-based BART fine-tunes filled statements such as “Bangkok is a location entity” for candidate spans and uses template scores to classify entities or non-entities.Non-entity spans use a separate negative template, “candidate_span is not a named entity.”
  • The method applies to arbitrary new entity categories without changing the output layer, while supporting continual learning.
  • Experiments cover both resource-rich CoNLL03 and few-shot settings, with competitive CoNLL03 performance and significant gains over prior methods in few-shot NER.

2 Related Work

Related work includes neural sequence-labeling systems, distance-based few-shot NER, and template-based language-model methods. This paper differs by using a language model to score each candidate span for sequence labeling rather than scoring only whole sentences or performing nearest-neighbor matching.

  • Neural NER methods use token-level classification, CRF decoding, sequence-to-sequence frameworks, label attention, Bayesian networks, or entity-aware pre-training.
  • Distance-based few-shot NER retrieves similar labeled instances or estimates label dependencies to transfer labels across domains.
  • Template-based NLP methods exploit pretrained language models by rephrasing inputs as cloze questions, generating label words, or constructing auxiliary sentences.
  • Unlike earlier template methods that score whole sentences or address sentence-level tasks, this work assigns language-model scores to spans given an input sentence.The authors describe it as the first template-based method applied to sequence labeling.

3 Background

The paper defines few-shot NER as adaptation from a labeled resource-rich dataset to a much smaller target dataset whose label vocabulary may differ. It contrasts traditional sequence labeling, whose output parameters depend on the label set, with the resulting adaptation limitations.

  • 3.1 Few shot Named Entity Recognition: Few-shot NER combines a resource-rich dataset with many labeled sequences and a low-resource dataset containing far fewer labeled sequence pairs.The target label vocabulary may differ from the resource-rich label vocabulary.
  • 3.1 Few shot Named Entity Recognition: The goal is to train an accurate and robust NER model for the low-resource domain using both resource-rich and low-resource data.
  • 3.2 Traditional Sequence Labeling Methods.: Traditional sequence labeling represents each output as a segmentation tag such as B, I, or O combined with an entity type.For example, B-person marks the first word of a person entity, while I-location marks an internal location token.
  • 3.2 Traditional Sequence Labeling Methods.: Traditional methods estimate each token label with an output layer whose trainable parameters depend on the label vocabulary.The model uses BERT and BART encoders to learn sequence representations.
  • 3.2 Traditional Sequence Labeling Methods.: When source and target label sets differ, the target output layer must be trained from scratch, limiting label-association transfer and preventing direct zero-shot use.

4 Template-Based Method

The method formulates NER as sequence-to-sequence language-model ranking: BART scores templates filled with candidate spans and entity labels. It transfers to new domains by replacing template label words while retaining the trained model and algorithms.

  • 4 Template-Based Method: NER is treated as a language-model ranking problem in a sequence-to-sequence framework, with input text as source and filled templates as targets.Templates contain a candidate span and an entity-type label.
  • 4.1 Template Creation: Templates map each label to a natural word and include positive entity and negative non-entity forms.The positive template expresses an entity type, while the negative template states that the span is not a named entity.
  • 4.2 Inference: Inference enumerates candidate spans, restricts spans to one-to-eight-gram n-grams, and scores the resulting templates with fine-tuned BART.Each sentence produces 8n templates under this efficiency restriction.
  • 4.2 Inference: Overlapping spans with different labels are resolved by selecting the span-label assignment with the higher template score.This rule avoids contradictory predictions because the datasets contain no nested entities.
  • 4.3 Training: Training pairs each input with templates filled by gold entities and adds negative pairs from randomly sampled non-entity spans.The method constructs positive and negative sequence pairs for fine-tuning.
  • 4.3 Training: BART is trained with decoder cross-entropy between its output distribution and the original template.The encoder processes the input sentence, while decoder attention uses encoder representations and previous output tokens.
  • 4.4 Transfer Learning: For a new domain, the method fills templates with the new label set while leaving the rest of the model and algorithms unchanged.Few-shot target-domain sequence pairs are used to fine-tune the model trained on the rich-source domain.
  • 4.4 Transfer Learning: Label correlations such as person–character and location–city can enhance transfer learning across domains.The transfer process is described as low cost while preserving the trained model and algorithms.

5 Experiments

Experiments evaluate template-based BART in resource-rich, in-domain few-shot, cross-domain few-shot, frequency-stratified, continual-learning, and error-analysis settings. The method is competitive on CoNLL03 and consistently improves few-shot transfer, while label-semantic distance remains a limitation.

  • Template selection: 95.27% F1 is achieved by the best manual template, compared with 76.80% for an alternative template on CoNLL03 development data.The selected template is “⟨candidate_span⟩is a ⟨entity_type⟩entity”.
  • Standard NER setting: 92.55% F1 is achieved by ensembling three templates on CoNLL03, while template-based BART outperforms sequence-labeling BART by 1.30% absolute F1.The ensemble also increases precision by 1.21%.
  • In-domain few-shot setting: 11.26 and 12.98 F1 points separate template-based BART from BERT on low-resource CoNLL03 entity types, with similar performance on resource-rich types.The evaluated low-resource categories are LOC and MISC in the reported passage.
  • Cross-domain few-shot setting: 57.1% F1 is obtained with 20 instances per entity type on MIT Restaurant, exceeding BERT trained with 100 instances per entity type.This comparison is reported for training from scratch without source-domain data.
  • Cross-domain few-shot setting: 6.6, 6.9, and 5.4 average F1-point gains are reported on MIT Restaurant, MIT Movie, and ATIS, versus 3.1, 1.9, and 4.3 for BERT as labeled data increases.The method improves more with additional target-domain labeled data than distance-based methods, whose performance remains nearly unchanged.
  • Error analysis: 84.81 F1 is achieved for ACTOR when its embedding resembles PERSON, whereas SONG reaches only 34.97 F1 when its embedding is far from existing CoNLL03 labels.The passage attributes these differences to reliance on label semantics across domains.

6 Conclusion

The paper presents BART-based template NER as a stronger approach for few-shot settings with new entity categories. Experiments show competitive rich-resource performance and significant gains across cross-domain and few-shot benchmarks.

  • Template-based BART can be fine-tuned directly for target domains when new entity categories exist.
  • The method achieves competitive results on a rich-resource NER benchmark.
  • The model significantly outperforms traditional sequence-labeling and distance-based methods on cross-domain and few-shot NER benchmarks.
Loading 2106.01760v1…