PaperlayerView exact revision on arXiv

Source-linked AI summary

Template-free Prompt Tuning for Few-shot NER

Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, Xuanjing Huang

arXiv:2109.13532v3cs.CLcs.AI

TL;DR

Few-shot NER is challenging for template-based prompting because obtaining token labels requires enumerating potential entity spans, making decoding increasingly costly. EntLM reformulates NER as template-free language modeling with label-word prediction at entity positions and automated label-word selection. The method improves few-shot NER over BERT-tagger and template-based prompting while achieving decoding speeds up to 1930.12 times faster than the template-based method.

  • Problem

    Template-based prompting is difficult and time-consuming for token-level NER because it requires span enumeration and grows costlier with sentence length.

  • Method

    EntLM reformulates NER as an Entity-oriented LM task that predicts class-related label words at entity positions without templates and explores automated label-word selection.

  • Results

    EntLM significantly improves few-shot NER over BERT-tagger and template-based methods, with decoding up to 1930.12 times faster than the template-based method.

  • Takeaways & Limitations

    The method provides a template-free few-shot NER approach that retains language-model prediction and avoids span-enumerating template decoding.

  • Takeaways & Limitations

    The study focuses on few-shot NER with only K examples per class and does not assume rich-resource source-domain data or test-time support sets.

Abstract

from arXiv · show

Prompt-based methods have been successfully applied in sentence-level few-shot learning tasks, mostly owing to the sophisticated design of templates and label words. However, when applied to token-level labeling tasks such as NER, it would be time-consuming to enumerate the template queries over all potential entity spans. In this work, we propose a more elegant method to reformulate NER tasks as LM problems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models to predict a class-related pivot word (or label word) at the entity position. Meanwhile, we also explore principled ways to automatically search for appropriate label words that the pre-trained models can easily adapt to. While avoiding complicated template-based process, the proposed LM objective also reduces the gap between different objectives used in pre-training and fine-tuning, thus it can better benefit the few-shot performance. Experimental results demonstrate the effectiveness of the proposed method over bert-tagger and template-based method under few-shot setting. Moreover, the decoding speed of the proposed method is up to 1930.12 times faster than the template-based method.

1 Introduction

Prompt-based learning benefits few-shot classification through language-model objectives and template or label-word design, but template-based prompting is difficult and inefficient for token-level NER. EntLM removes templates, predicts label words at entity positions, and reports stronger few-shot performance with much faster decoding.

  • 1 Introduction: Template-based prompting is difficult to adapt to NER because span-level template search expands the search space and can overfit with few annotations.NER requires token- or span-level decisions rather than a single sentence-level classification query.
  • 1 Introduction: 21 queries are required to obtain all entities in the example sentence “Obama was born in America” using span-enumerating template prompts.The approach queries the language model separately for every candidate span.
  • 1 Introduction: Template-based NER decoding becomes increasingly time-consuming as sentence length grows, making it impractical for document-level corpora.The decoding burden arises from enumerating all possible spans.
  • 1 Introduction: EntLM reformulates NER as an Entity-oriented LM task without templates, predicting class-related pivot words at entity positions and original words elsewhere.The method preserves the language-model output head and introduces no new fine-tuning parameters.
  • 1 Introduction: Automatic label-word engineering selects discrete or virtual label words using lexicon statistics, language-model predictions, or prototypes.The goal is to find label words that represent the words belonging to each entity class.
  • 1 Introduction: 1930.12 times faster decoding is reported for the proposed method than for the template-based baseline, alongside verified few-shot effectiveness.The result is presented as an experimental comparison under the few-shot setting.

2 Problem Setup

The paper studies few-shot NER with only a small number of labeled examples per class and evaluates on an unseen test set from the task distribution.

  • 2 Problem Setup: Few-shot NER assumes only K training examples for each class, totaling Ktot = K × |Y| labeled examples.The setting avoids assuming a rich-resource source domain or support sets during testing.
  • 2 Problem Setup: The model is tested on an unseen test set (Xtest, Y test) sampled from Dtest after fine-tuning on the limited training data.For NER, each training sample is a contiguous entity span labeled with a positive class.

3 Approach

The approach reformulates few-shot NER as template-free language modeling, predicting class-related label words at entity positions while reusing the pretrained LM head. It also introduces automatic label-word search and one-pass decoding to avoid span enumeration.

  • 3.1 Template-based Prompt Tuning: Template-based NER requires span-level template queries, making template search difficult and decoding increasingly time-consuming with sentence length.The method must enumerate all possible spans to obtain token labels, limiting practicality for longer inputs.
  • 3.2 Entity-Oriented LM Fine-tuning: EntLM reuses the pretrained LM head and predicts class-related label words at entity positions instead of replacing the head with a classifier.None-entity positions retain their original words, while entity tokens are replaced by mapped label words in the target sentence.
  • 3.2 Entity-Oriented LM Fine-tuning: EntLM trains the model to maximize P(XEnt|X), where entity tokens in XEnt are replaced by their mapped label words and non-entity tokens remain unchanged.The objective preserves a language-modeling formulation while encoding NER labels at entity positions.
  • 3.2 Entity-Oriented LM Fine-tuning: One-pass decoding obtains all token labels directly from the test sentence, making inference more efficient than template-based prompt querying.The class probability for token i is derived from the LM probability of the corresponding mapped label word.
  • 3.3 Label Word Engineering: Label-word search selects class pivots from data frequencies, LM output distributions, or both, while virtual label words use class-wise mean vectors as prototypes.Unlabeled data and lexicon-based annotation are explored to make selection more consistent, although lexicon annotation is noisy.

4 Experiments

The experiments evaluate EntLM for few-shot NER across datasets, baselines, label-word selection, lexicon and unlabeled-data sizes, and further pre-training. EntLM consistently outperforms the baselines while offering substantially faster decoding and robustness to limited auxiliary resources.

  • Experimental Setup: Experiments use K=5, 10, 20, and 50 examples per class, with three sampled training sets and four repetitions per set.The evaluation covers three NER datasets from newswire, general, and review domains.
  • Few-shot Results: EntLM consistently outperforms all baselines across the three datasets and few-shot settings, with especially strong results in 5-shot learning.Its performance is also reported as more stable according to deviation.
  • Few-shot Results: EntLM consistently outperforms the template-based TemplateNER method and can also outperform StructShot when using a Viterbi decoder.The metric-based NNShot method does not show advantages over BERT-tagger when no rich-resource source domain is available.
  • Efficiency Study: 1930.12 times faster is EntLM’s maximum decoding-speed advantage over TemplateNER, while EntLM has comparable speed to BERT-tagger.EntLM requires only one token-classification pass per batch for decoding.
  • Analysis: Data&LM+Virtual label-word selection performs best among the virtual-selection methods, and EntLM remains highly performant with 5% of the original lexicon.The method is also reported as robust to using only 5%–80% of the original unlabeled dataset.
  • Further Pre-training: Further pre-training on domain-specific unlabeled data substantially boosts EntLM’s few-shot ability but is less helpful for classifier-based fine-tuning.The analysis attributes this difference to EntLM’s LM objective benefiting from a task-specific output distribution.

5 Related Works

Prior work applies prompts and metric-based methods to few-shot NER, but often relies on episode training or rich-resource source domains. This paper instead adopts a practical setting with only a few labeled samples per class and adapts prior methods as baselines.

  • Prompt-based Learning: Prompt-based learning reformulates downstream tasks as cloze tasks using textual templates and label words, whose design strongly affects few-shot performance.Prior studies explored manual and automated template construction.
  • Few-shot NER: Few-shot NER studies include prototypical networks, nearest-neighbor prediction, comprehensive baseline evaluations, and methods for mining undefined classes.Many of these studies use episode training or assume a rich-resource source domain.
  • Positioning: This work follows a more practical few-shot setting with only a few samples per class for training and adapts previous methods as competitive baselines.The setting is based on Gao et al. (2021).

6 Conclusion

The paper concludes that EntLM provides template-free prompt tuning for few-shot NER by predicting label words at entity positions. It reports improved few-shot performance and much faster decoding than the template-based method.

  • Conclusion: EntLM reformulates few-shot NER as an Entity-oriented LM task that predicts label words at entity positions during fine-tuning.This removes the need for complicated template-based prompting.
  • Conclusion: EntLM’s LM objective reduces the gap between pre-training and fine-tuning, and experiments show significant improvement over BERT-tagger and template-based methods.The conclusion presents this as the basis for improved few-shot NER performance.
  • Conclusion: 1930.12 times faster is EntLM’s maximum decoding-speed advantage over the template-based method.This result is reported as part of the conclusion’s efficiency comparison.

A.1 Implementation Details

The implementation uses a BERT-base model and fixed training settings for few-shot experiments. Label words are selected with the Data&LM+Virtual procedure under a fixed conflict threshold and vocabulary size.

  • Model and Training: The method is implemented with Hugging Face Transformers and uses bert-base-cased for all experiments except TemplateNER.The proposed method introduces no new parameters.
  • Model and Training: Few-shot training uses learning rate 1e-4, batch size 4, 20 epochs, AdamW, and the same linear-decay schedule as pre-training.These settings apply to the BERT-base baselines and the proposed method.
  • Label-word Selection: Label words are selected with Data&LM+Virtual using conflict ratio Th = 0.6 and the top k=6 high-frequency words.The same selection configuration is used across datasets.

A.2 Sampling Algorithm

The paper uses an exact sampling algorithm to obtain K samples for each class in few-shot NER, rather than relying on greedy sampling. Across all three datasets, it obtains the required per-class sample count under every K-shot setting.

  • A.2 Sampling Algorithm: The algorithm samples exactly K examples for each class in the training set.It differs from the greedy sampling method used in previous methods.
  • A.2 Sampling Algorithm: The procedure tracks each entity class and rejects samples that would exceed the K-shot limit.It computes the number of mentions for each class before adding a sample.
  • A.2 Sampling Algorithm: All three datasets yield exactly K samples per class under every evaluated K-shot setting.This verifies that the sampling procedure satisfies the intended few-shot constraint.

A.3 Effect of Conflict threshold

The conflict threshold affects label-word quality and few-shot performance differently across the two methods. Increasing the threshold improves Data&LMSearch, whereas Data&LM+Virtual is less sensitive because it selects multiple words.

  • A.3 Effect of Conflict threshold: Lower conflict thresholds produce improper label words and noisy annotated entities for Data&LMSearch.The resulting noise is associated with weaker performance at lower thresholds.
  • A.3 Effect of Conflict threshold: Performance improves as the conflict threshold increases for Data&LMSearch.The passage attributes this trend to reducing improper label words and noisy annotations.
  • A.3 Effect of Conflict threshold: Data&LM+Virtual is less affected by conflict words because it selects multiple words to construct a virtual vector.The multiple-word construction reduces the significance of individual conflict words.

A.4 Effect of k in virtual method

The virtual method is evaluated by varying its top k setting in CoNLL 5-shot experiments. Its performance remains robust for k >= 3, and the main experiments use k = 6 across datasets.

  • A.4 Effect of k in virtual method: The study evaluates the choice of top k for the virtual method using Data&LMSearch+Virtual on CoNLL 5-shot data.Figure 7 reports the impact of this hyperparameter choice.
  • A.4 Effect of k in virtual method: Performance is robust to k and consistently good when k >= 3.The result indicates that the method does not depend strongly on the precise top-k choice in this experiment.
  • A.4 Effect of k in virtual method: The main experiments choose k = 6 for all datasets.This setting is selected after observing robust performance for sufficiently large k.
Loading 2109.13532v3…