PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-NER: Named Entity Recognition via Large Language Models

Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang

arXiv:2304.10428v4cs.CL

TL;DR

LLMs underperform supervised systems on NER because generation does not directly match token-level sequence labeling. GPT-NER reformulates NER as marked-text generation and adds self-verification, achieving supervised-comparable results across five datasets and stronger performance in extremely low-resource settings.

  • Problem

    LLMs perform below supervised NER baselines because NER is token-level sequence labeling while LLMs are text-generation models.

  • Method

    GPT-NER marks entities with special tokens in generated text and uses self-verification to check extracted entities against labeled tags.

  • Results

    GPT-NER achieves comparable performance to fully supervised baselines across five NER datasets and performs significantly better than supervised models when labeled data is extremely scarce.

  • Takeaways & Limitations

    GPT-NER demonstrates potential for real-world NER applications when the quantity of labeled samples is limited.

  • Takeaways & Limitations

    Sentence-level retrieval can miss token-level NER evidence, and nested NER is harder when similar entity types must be distinguished from limited demonstrations.

Abstract

from arXiv · show

Despite the fact that large-scale Language Models (LLM) have achieved SOTA performances on a variety of NLP tasks, its performance on NER is still significantly below supervised baselines. This is due to the gap between the two tasks the NER and LLMs: the former is a sequence labeling task in nature while the latter is a text-generation model. In this paper, we propose GPT-NER to resolve this issue. GPT-NER bridges the gap by transforming the sequence labeling task to a generation task that can be easily adapted by LLMs e.g., the task of finding location entities in the input text "Columbus is a city" is transformed to generate the text sequence "@@Columbus## is a city", where special tokens @@## marks the entity to extract. To efficiently address the "hallucination" issue of LLMs, where LLMs have a strong inclination to over-confidently label NULL inputs as entities, we propose a self-verification strategy by prompting LLMs to ask itself whether the extracted entities belong to a labeled entity tag. We conduct experiments on five widely adopted NER datasets, and GPT-NER achieves comparable performances to fully supervised baselines, which is the first time as far as we are concerned. More importantly, we find that GPT-NER exhibits a greater ability in the low-resource and few-shot setups, when the amount of training data is extremely scarce, GPT-NER performs significantly better than supervised models. This demonstrates the capabilities of GPT-NER in real-world NER applications where the number of labeled examples is limited.

1 Introduction

GPT-NER adapts LLMs to NER by reformulating sequence labeling as text generation and adding self-verification against hallucinated entities. Across five datasets, it matches supervised baselines and is especially effective with scarce labeled data.

  • Motivation: LLM performance on NER remains below supervised baselines because sequence labeling and text generation impose different task formulations.NER assigns entity-type labels to tokens, whereas LLMs generate text sequences.
  • GPT-NER: GPT-NER marks entities with special tokens while copying the remaining input, transforming NER into an LLM-compatible generation task.For example, Columbus is a city becomes @@Columbus## is a city for location extraction.
  • Self-verification: Self-verification prompts the LLM to check whether extracted entities belong to labeled tags, countering overprediction of entities in NULL inputs.The strategy is applied after entity extraction and is reported to improve performance.
  • Evaluation: GPT-NER achieves comparable performance to fully supervised baselines across five flat and nested NER datasets.The authors describe this as the first such result to their knowledge.
  • Low-resource performance: With extremely scarce training data, GPT-NER significantly outperforms supervised models, including about 60 F1 from 8 training examples versus around 0.The paper presents this as evidence of stronger low-resource generalization and potential use when labeled samples are scant.

2 Related Work

Prior NER systems commonly use token representations and classifiers, while LLM research has expanded through fine-tuning and in-context learning. The related work positions GPT-NER at the intersection of these traditions.

  • NER methods: Traditional NER approaches formulate the task as sequence labeling using LSTMs, CNNs with CRF, character CNNs, BERT, MRC, or GNN-based models.These methods obtain token-level or word-level representations and classify or decode them into entity labels.
  • LLM adaptation: LLMs have produced performance gains across diverse NLP tasks, with downstream adaptation commonly divided into fine-tuning and in-context learning.Fine-tuning continues training on supervised downstream data.
  • In-context learning: In-context learning prompts LLMs to generate outputs from few-shot demonstrations and has been studied across translation and other downstream tasks.The cited work includes GPT-3 analyses and demonstrations-based reformulations of tasks.

3 Background

NER identifies and classifies entities by assigning predefined labels to words in a sentence. A conventional sequence-labeling pipeline extracts token representations and then classifies them.

  • Task definition: NER assigns each word x_i in a sentence X an entity type y from a predefined label set Y.The sentence contains n words, and Y denotes the available entity labels.
  • Pipeline: Sequence-labeling NER decomposes into representation extraction followed by classification.The first stage obtains token representations; the second predicts entity-label distributions.
  • Representation extraction: Representation extraction encodes the input sentence and uses each word’s final-layer embedding h_i as its high-dimensional token representation.The embedding dimension is m, while n denotes sentence length.
  • Classification: Classification sends each token representation through a multilayer perceptron and softmax to produce a distribution over the named-entity vocabulary.This converts token embeddings into entity-label predictions.

4 GPT-NER

GPT-NER adapts LLMs to NER by constructing prompts and converting sequence labeling into a text-generation format. It combines few-shot demonstrations, entity-level retrieval, and self-verification to guide extraction and reduce hallucinated entities.

  • GPT-NER Framework: GPT-NER constructs prompts, generates a labeled text sequence, and transforms that sequence into entity labels.The framework comprises prompt construction, LLM generation, and conversion of generated text into final entity labels.
  • Prompt Construction: The prompt’s task description specifies the entity category, while demonstrations and the input sentence complete the prompt.The entity-type instruction can turn an N-class task into N binary classification tasks because each input is processed for each entity label.
  • Prompt Construction: Few-shot demonstrations regulate output formatting and provide task-specific references for predictions.Demonstrations contain paired input and output sequences and are appended to the prompt.
  • Output Format: The marking format addresses alignment and length difficulties that make direct label-sequence generation difficult for LLMs.Direct formats require alignment between input positions and labels, and GPT-3 struggles to generate outputs matching long input lengths.
  • Output Format: GPT-NER marks entities with @@ and ## while copying the remaining input text, avoiding token-by-token label generation.For example, “Columbus is a city” becomes “@@Columbus## is a city”; entity-free inputs are copied unchanged.
  • Demonstration Retrieval: Entity-level retrieval uses recognized entity embeddings as kNN queries over a datastore of extracted entity–sentence pairs.This targets token-level evidence rather than sentence-level similarity, which may retrieve semantically similar sentences lacking useful NER evidence.
  • Self-Verification: Self-verification asks whether each extracted entity belongs to the specified entity type, countering LLM overprediction of entities in NULL inputs.The verification step follows entity extraction and is reported to improve performance.

5 Experiments

Experiments evaluate GPT-NER on flat and nested NER, showing that retrieval quality and self-verification substantially affect performance. GPT-NER is competitive with supervised baselines and particularly strong when labeled data are scarce, though gaps remain on nested NER.

  • Experimental Setup: GPT-NER is evaluated on flat CoNLL2003 and OntoNotes5.0 using span-level precision, recall, and F1, with comparisons against established NER systems.The flat-NER experiments include sampled and full test sets.
  • 5.1.1 Results on Flat NER: 84.36 and 75.32 rise to 89.97 and 81.73 on CoNLL2003 and OntoNotes5.0 when sentence-level retrieval is replaced by token-level retrieval.Token-level representations better match NER’s local, entity-focused nature and retrieve demonstrations associated with specific entity types.
  • 5.1.1 Results on Flat NER: 89.97 rises to 90.62 for zero-shot learning and 84.97 rises to 85.91 for few-shot learning after adding self-verification on full CoNLL2003.The authors attribute this improvement to alleviating GPT-3’s overprediction of entities.
  • 5.1.1 Results on Flat NER: GPT-NER reaches 90.91 versus BERT’s 92.8 on CoNLL2003 and 82.20 versus 89.16 on OntoNotes5.0, while remaining below supervised SOTA results.The reported SOTA comparisons are 94.6 versus 90.91 on CoNLL2003 and 92.07 versus 82.20 on OntoNotes5.0.
  • 5.1.2 Results on Nested NER: 48.4 rises to 73.62 on ACE2004 with entity-level kNN retrieval, while entity-level retrieval also improves results from 60.68 to 73.62 on ACE2004 and 56.68 to 69.06 on GENIA.Adding self-verification further raises ACE2004 sentence-level results from 60.68 to 62.31 in zero-shot and 62.52 in few-shot learning.
  • 5.2.1 Results: At 10% of the data, supervised performance improves significantly while GPT-3 improves marginally, emphasizing retrieved-demonstration quality and prompt structure.The paper gives kNN retrieval and self-verification as examples of improving demonstration quality and prompt structure.

6 Ablation Study

The ablations compare output formats, demonstration retrieval, and demonstration quantity for GPT-NER. The proposed entity-marking format and increasing demonstrations improve performance, while sparse retrieval can fail when examples lack entities.

  • Output formats: GPT-NER uses special tokens @@ and ## to surround recognized entities in the generated output.For example, “@@Columbus## is a city” marks Columbus as an entity.
  • Demonstration quantity: As the number of demonstrations increases, all three LLM-based results keep rising and have not plateaued near GPT-3’s 4,096-token limit.The authors therefore report remaining room for improvement with more demonstrations.
  • Demonstration retrieval: With only k = 2 or 4 demonstrations, kNN retrieval can underperform random retrieval when retrieved examples contain no entities.Such examples may fail to provide the output-format information needed for entity extraction.

I am an excellent linguist. The task is to label

The examples illustrate how the task asks the model to identify organization entities in short sentences. The figure compares results while varying the number of k-shot demonstrations.

  • Task examples: The examples present sentences about Korean pro-soccer games, Australia, and Japan, with outputs that initially preserve the input text.The Japan example additionally labels Japan as an organization entity.
  • Task examples: The Japan example marks Japan as an organization entity in the output.The output places “[Organization Entity]” after Japan.
  • Demonstration comparison: Figure 5 compares GPT-NER results under different numbers of k-shot demonstrations.The supplied caption identifies the comparison as varying k-shot demonstrations.

7 Conclusion

GPT-NER adapts LLMs to NER by converting sequence labeling into labeled text generation and adding self-verification. Across flat and nested datasets, it matches supervised baselines and is stronger in extremely low-resource settings.

  • Conclusion: GPT-NER surrounds entities with special tokens to generate labeled sequences, bridging NER’s sequence-labeling formulation and LLM text generation.The conclusion describes this as the approach for adapting LLMs to NER.
  • Conclusion: GPT-NER adds self-verification to alleviate LLM hallucination during NER.The strategy prompts the model to check whether extracted entities belong to a labeled entity tag.
  • Conclusion: GPT-NER achieves comparable performance to fully supervised baselines and performs significantly better than supervised models when labeled data is extremely scarce.The reported experiments cover both flat and nested NER datasets.

A Datasets

The evaluation uses four datasets spanning general-domain flat NER and nested NER, including ACE2004, ACE2005, and GENIA. Their entity inventories and domains differ substantially.

  • Flat NER datasets: CoNLL2003 contains four entity types: Location, Organization, Person, and Miscellaneous.The dataset description references annotation details and corpus statistics in Tables 4 and 6.
  • Flat NER datasets: OntoNotes5.0 contains 18 named-entity types.Its entity annotations and corpus statistics are listed in Tables 5 and 6.
  • Nested NER datasets: ACE2004 and ACE2005 are English nested NER datasets containing seven entity types, including GPE, ORG, PER, FAC, VEH, LOC, and WEA.The datasets include nested-entity annotations and corpus statistics.
  • Nested NER datasets: GENIA is an English nested NER dataset in molecular biology with five entity types: cell line, cell type, DNA, RNA, and protein.Entities are named according to their biological meaning.

B Error Cases of Format BMES and Entity-position

On sample-100 CoNLL2003 examples, GPT-3 struggles with both BMES and entity-position output formats. BMES generation becomes difficult for long inputs, while entity-position generation produces incorrect position information.

  • Dataset and annotations: The examples use English CoNLL2003, whose annotations include organization, person, location, and miscellaneous entity types.The dataset tables describe these four annotation categories.
  • BMES format: GPT-3 struggles to generate BMES outputs with the same length as long input sentences.The examples identify output-length mismatch as a particular difficulty for longer sentences.
  • Entity-position format: GPT-3 is confused when generating the correct position information in the entity-position format.The observed errors concern the position information required by this output format.

C Examples

The examples compare random, sentence-level, and entity-level retrieval for GPT-NER demonstrations. Entity-level retrieval selects examples focused on the same local entities as the input, making GPT-3 prediction easier and emphasizing demonstration quality.

  • Example sets: Tables 11–13 present CoNLL2003 examples for random retrieval, sentence-level embedding, and entity-level embedding, respectively.The examples are described as demonstrations of GPT-NER.
  • Random retrieval: Random retrieval gives all sentences the same opportunity to appear as few-shot examples, usually without similarity to the input sentence.The retrieved examples generally do not contain similar examples to the input.
  • Sentence-level embedding: Sentence-level embedding retrieves semantically similar examples, but they may not focus on the same local entities as the input.Semantic similarity does not necessarily align the local entity content.
  • Entity-level embedding: Entity-level embedding retrieves examples focused on the same local entities as the input, leading GPT-3 prediction more easily.The passage connects this observation to the effectiveness of demonstration quality in in-context learning.
Loading 2304.10428v4…