PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER

Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, Weiran Xu

arXiv:2203.03903v1cs.CL

TL;DR

Few-shot NER is difficult because token-label annotations are scarce and existing prompt templates are mostly designed for sentence-level tasks. InstructionNER uses instruction-conditioned generation with entity-extraction and entity-typing tasks. It consistently outperforms other baselines in few-shot settings across several NER datasets.

  • Problem

    Few-shot NER is challenging because token-label annotations are scarce and costly, while most prompt templates are designed for sentence-level tasks rather than sequence labeling.

    Source: §1 · pp. 1–2

  • Method

    InstructionNER reformulates NER as generation from source sentences enriched with task instructions and entity-type options, adding entity extraction and entity typing as auxiliary tasks.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    InstructionNER consistently outperforms other baselines in few-shot settings; under 10-shot, its best implementation improves F1 over LightNER by 24.4% on MIT Movie, 10.9% on MIT Restaurant, and 13.1% on ATIS.

    Source: §4.2 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    The reported results support InstructionNER's effectiveness for few-shot NER across several datasets.

    Source: §6 · p. 9

    On 5ATIS, which has 79 entity types, using natural-language labels directly would make the input instruction too long.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recently, prompt-based methods have achieved significant performance in few-shot learning scenarios by bridging the gap between language model pre-training and fine-tuning for downstream tasks. However, existing prompt templates are mostly designed for sentence-level tasks and are inappropriate for sequence labeling objectives. To address the above issue, we propose a multi-task instruction-based generative framework, named InstructionNER, for low-resource named entity recognition. Specifically, we reformulate the NER task as a generation problem, which enriches source sentences with task-specific instructions and answer options, then inferences the entities and types in natural language. We further propose two auxiliary tasks, including entity extraction and entity typing, which enable the model to capture more boundary information of entities and deepen the understanding of entity type semantics, respectively. Experimental results show that our method consistently outperforms other baselines on five datasets in few-shot settings.

1 Introduction

Few-shot NER is challenging because token-level annotations are scarce and costly, while prompt templates designed for sentence-level tasks do not fit sequence labeling. InstructionNER reformulates NER as instruction-conditioned generation and adds entity extraction and typing tasks.

1 Introduction

  • Scarce, costly token-label annotations make few-shot NER challenging, and most existing prompt templates do not fit sequence-labeling tasks.Prompt-based methods have shown gains in low-resource scenarios, motivating a reformulation better suited to NER.Source: §1 · pp. 1–2
  • Cui et al. (2021) enumerate candidate spans for template-based generation, while BARTNER (Yan et al., 2021) predicts entities and type indexes, and LightNER adds soft prompts.These approaches motivate asking whether heuristic prompts in the source sentence can better use pretrained semantic knowledge for low-resource NER.Source: §1 · p. 2
  • InstructionNER reformulates NER as natural-language generation, adding task instructions and candidate entity-type options to the source sentence and generating entity names with their types.The framework uses T5 and natural-language outputs.Source: §1 · p. 2
  • Entity extraction targets entity boundaries, while entity typing targets type semantics; the paper reports effectiveness on three benchmarks, especially in data-scarce scenarios.The two auxiliary tasks are introduced alongside the instruction-based generation framework.Source: §1 · p. 2

2 Related Works

Related work contrasts sequence-labeling and generative formulations of NER, then reviews prompt-based methods and their use in few-shot NER. InstructionNER places task instructions and answer options in the source sentence.

2.1 Named Entity Recognition

  • Sequence labeling with token classifiers or CRFs is suited to flat NER but less suited to other NER subtasks.BARTNER reformulates three NER subtasks as generation and predicts entities with type indexes; this paper instead uses T5 to generate entity names and types in natural language.Source: §2.1 · p. 2

2.2 Prompt-based Methods

  • Prompt-based learning seeks to use pretrained knowledge for few-shot performance, while instruction-based methods train models on task descriptions and answer options.The paper adapts this idea with templates describing each task and candidate entity categories.Source: §2.2 · p. 3
  • Template-based BART evaluates candidate spans, whereas LightNER uses continuous prompts; InstructionNER instead puts task instructions and answer options in the source sentence.Enumerating candidate spans is described as time-consuming.Source: §2.2 · p. 3

3 Methodology

InstructionNER casts NER as T5-based generation: inputs combine a sentence, task instruction, and entity-type options, while outputs express entities and types in natural language. Joint entity-extraction and entity-typing tasks target boundaries and type semantics, and generated text is parsed into NER triplets.

Problem Definition

  • NER represents each entity occurrence as a triplet of left boundary, right boundary, and entity type within a sentence.The span from the left through right boundary is inclusive.Source: §3 · p. 3

Solving NER through T5

  • The main task fine-tunes T5 to generate entity names and types in natural language from a sentence enriched with an instruction and candidate entity-type options.The options act as a hint and constraint on which types to recognize.Source: §3 · p. 3
  • Entity types can be represented with natural-language labels or synthetic tokens, whose embeddings are randomly initialized during fine-tuning.The paper compares both strategies in analysis experiments.Source: §3 · pp. 3–4

Auxiliary Tasks

  • Jointly trained entity extraction simplifies NER by generating entity spans without assigning types, targeting improved span F1 and main-task NER accuracy.Its input instruction omits the options field, and its output contains only entity spans.Source: §3 · p. 4
  • Entity typing supplies entity occurrences and trains the model to predict their categories, focusing on type semantics while maintaining entity-occurrence generation.The authors state that this is intended to improve NER performance.Source: §3 · p. 4

Evaluation

  • At evaluation, generated text is split into comma-separated clauses; clauses matching the “is a/an” pattern are aligned to the sentence and type set to form triplets.Clauses that fail matching are ignored; F1 is the main metric.Source: §3 · pp. 4–5

4 Experiments

The experiments evaluate InstructionNER and its variants on low-resource datasets using few-shot settings and compare them with established NER baselines. InstructionNER outperforms the compared baselines, while ablations examine the contributions of instructions, options, and auxiliary tasks.

4.1 Setup

  • Experiments use CoNLL-2003 as a rich-resource dataset and MIT Movie, MIT Restaurant, and ATIS as low-resource datasets, with greedy sampling to target k instances per entity type.When an entity occurs fewer than k times in training, all its occurrences are sampled; each few-shot setting is sampled three times and results are averaged with standard deviations.Source: §4.1 · p. 5

4.2 Main Results

  • Across 10/20/50-shot settings, InstructionNER methods consistently outperform the compared baselines; at 10-shot, the best implementation improves F1 by 24.4%, 10.9%, and 13.1% over LightNER on MIT Movie, MIT Restaurant, and ATIS, respectively.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Ablation Analysis

  • Removing instructions and options while keeping the output template yields the T5-natural baseline; InstructionNER outperforms it across data sizes, with the gap narrowing as training data increases.Source: §4.3 · p. 6, Figure 3 · p. 6
  • On MIT Movie and MIT Restaurant, ET improves F1 score(%) by 0.4/1.5/1.1 and EE by 1.2/1.2/0.6 on average under 10/20/50-shot settings.Source: §4.3 · p. 6
  • ET improves F1 more than span F1, whereas EE improves both in roughly similar proportion; combining ET and EE reduces performance, which the authors conjecture may reflect a 1:2 main-to-auxiliary data ratio.Source: Figure 4 · p. 6, §4.3 · p. 7

5 Analysis

The analysis compares backbones and transfer performance, then examines how entity-type wording performs across data regimes. It reports stronger few-shot results from InstructionNER across transfer settings, with label-word strategy outcomes varying by resource level.

5.1 Comparison of Different PLMs

  • T5-v1.1 and T5 implementations perform closely, while BART as the backbone performs very badly.The authors attribute the BART difference to its reconstruction pre-training objective and choose T5, whose corrupted-token prediction is closer to the task.Source: §5.1 · p. 7
  • T5-base InstructionNER outperforms Template based on BART-large despite using fewer parameters.The authors interpret this comparison as evidence that the instruction-based framework effectively mines knowledge in text-to-text PLMs.Source: §5.1 · p. 7

5.2 Transfer Analysis

  • The transfer experiments train on CoNLL03 and continue training on 10, 20, or 50 MIT Restaurant instances per entity type.Neigh.Tag., Example., and MP-NSP are included as transfer-setting baselines.Source: §5.2 · p. 7
  • InstructionNER outperforms the baselines in all three CoNLL03-to-MIT Restaurant transfer settings.Table 3 reports F1 scores for the transfer setting and brackets improvements over in-domain results.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Label Semantic Analysis

  • Synthetic entity-type tokens perform worse in few-shot settings but better than natural-language type words in rich-resource settings.The authors conjecture that natural-language label semantics help in low-resource settings, while synthetic tokens can specialize with enough supervision.Source: §5.3 · p. 8

6 Conclusion

InstructionNER reformulates NER as generation with descriptive instructions and limited answer options, supplemented by entity extraction and typing tasks. The paper concludes that it consistently outperforms baselines in few-shot settings, while noting a label-length constraint for ATIS.

6 Conclusion

  • InstructionNER turns sequence labeling into generation by adding descriptive task instructions and limited answer options to source sentences.The model uses pre-trained language-model semantic knowledge to generate entity words and types.Source: §6 · p. 8
  • ATIS has 79 entity types, making direct natural-language labels too long for the input instruction.This is the stated reason for the ATIS label-word exception.Source: §6 · p. 8
  • Entity extraction and entity typing auxiliary tasks target entity-boundary information and entity-type semantics, respectively.Experiments on several NER datasets consistently outperform other baselines in few-shot settings.Source: §6 · p. 9

A Implementation Details

The implementation uses T5 backbones and reports model sizes, hardware, training time, and software versions.

A Implementation Details

B Dataset Statistics

The appendix points to dataset statistics and a figure comparing F1 across entity-extraction and entity-typing data-sample ratios.

B Dataset Statistics

  • Table 4 is identified as containing statistics for the mentioned NER datasets.The supplied table caption does not specify individual statistics.Source: Table 4 · p. 10, B Dataset Statistics · p. 10
  • Figure 6 reports MIT Restaurant F1 for different entity-typing and entity-extraction sample ratios under 20- and 50-shot settings.Source: Figure 6 · p. 11

C Performance on Relatively Abundant Settings

On MIT Movie and MIT Restaurant at 100/200/500 shots, InstructionNER’s gains are small and it ultimately underperforms LightNER. The authors attribute its few-shot efficiency to general knowledge and meta-learning, while noting that this generality may limit specialization with more data.

C Performance on Relatively Abundant Settings

D The Ratio of Auxiliary Tasks

The balance between auxiliary-task and main-task data matters: combining entity extraction and typing can reduce NER performance when auxiliary data is excessive. In tested ratios, smaller auxiliary-task samples improve F1 over no auxiliary data or full auxiliary data.

D The Ratio of Auxiliary Tasks

  • The authors conjecture that combining entity extraction and typing reduces the main-to-auxiliary data ratio to 1:2, shifting focus from the main NER task.They offer this as an explanation for the observed performance drop, not as a proven cause.Source: D The Ratio of Auxiliary Tasks · p. 11
  • When auxiliary data is at most half the main-task amount, InstructionNER achieves better F1 than with no auxiliary data or full auxiliary data.The tested sample ratios are 0.1–0.5; the full-auxiliary condition is 1.Source: D The Ratio of Auxiliary Tasks · p. 11
  • When auxiliary-task data exceeds the main-task amount, performance drops by varying degrees and sometimes falls below the no-auxiliary condition.Source: D The Ratio of Auxiliary Tasks · p. 11

Related papers

Loading 2203.03903v1…