PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-Learning for Fine-Grained Entity Typing

Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, Hong-Gee Kim

arXiv:2108.10604v1cs.CLcs.AI

TL;DR

Fine-grained entity typing requires classifying entities into specific types across fully supervised, few-shot, and zero-shot settings, motivating better use of PLM knowledge when supervision is limited. The paper introduces prompt-based pipelines with entity-oriented verbalizers and templates, plus distribution-level self-supervision for zero-shot learning. Across three benchmarks, prompt-learning methods significantly outperform fine-tuning baselines, especially with insufficient training data.

  • Problem

    The paper investigates how to apply prompt-learning to fine-grained entity typing across fully supervised, few-shot, and zero-shot settings.

  • Method

    The method converts typing into masked language modeling with entity-oriented prompts, then uses distribution-level self-supervision on paired unlabeled examples for zero-shot learning.

  • Results

    Prompt-learning methods significantly outperform fine-tuning baselines across three benchmarks, especially when training data is insufficient.

  • Takeaways & Limitations

    Prompt-learning can exploit prior knowledge in PLMs for entity typing under fully supervised, few-shot, and zero-shot conditions.

Abstract

from arXiv · show

As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{cloze}-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on three fine-grained entity typing benchmarks (with up to 86 classes) under fully supervised, few-shot and zero-shot settings show that prompt-learning methods significantly outperform fine-tuning baselines, especially when the training data is insufficient.

1 Introduction

Prompt-learning reformulates downstream tasks as cloze-style masked-language problems to use knowledge in PLMs. This paper applies that approach to fine-grained entity typing across supervision regimes and introduces a self-supervised zero-shot strategy.

  • Prompt-learning motivation: Prompt-learning formalizes downstream tasks as cloze-style tasks, allowing PLMs to predict missing label words instead of using the original task formulation.This approach avoids extra neural layers and narrows the objective gap between pre-training and fine-tuning.
  • Prompt-learning motivation: Fine-grained entity typing classifies marked entities into specific types, and prompts can expose type information encoded in PLM vocabulary probabilities.For “New York,” the prompted model assigns higher probability to “city” than “person.”
  • Paper scope: The paper develops an entity-oriented prompt-learning pipeline for fully supervised, few-shot, and zero-shot fine-grained entity typing.The pipeline constructs prompts and treats typing as a cloze-style task.
  • Paper scope: For zero-shot typing, a self-supervised strategy optimizes similarity between predicted probability distributions of paired examples to summarize entity types.The strategy is designed for settings without explicit supervision.
  • Empirical evaluation: Experiments use FEW-NERD, OntoNotes, and BBN, whose rich type hierarchies require models to detect entity attributes.The authors report significant improvements, especially in zero-shot and few-shot settings.

2 Background

Entity typing predicts a marked entity’s type from context, while conventional fine-tuning uses a PLM representation and an added output layer. Prompt-based tuning instead converts classification into masked language modeling with label-word sets and templates.

  • Problem definition: Entity typing predicts a type y from a predefined set Y for a marked entity mention in each input sentence.The task is treated as context-aware classification because the same mention can receive different types in different contexts.
  • Vanilla fine-tuning: Vanilla fine-tuning feeds the PLM’s [CLS] representation into an output layer to predict the label distribution.The output-layer parameters and PLM parameters are tuned together.
  • Prompt-based tuning: Prompt-based tuning assigns each entity type a label-word set and adds a template containing [MASK] to the input.The model predicts a vocabulary word associated with the target type.
  • Prompt-based tuning: Prompt-learning transfers classification into a masked language modeling problem by predicting the missing label word.This reframing enables prompts to extract and aggregate entity-type information from PLMs.

3 Prompt-learning for Entity Typing: A Naive Pipeline

The naive prompt-learning pipeline reformulates fine-grained entity typing as masked language modeling using label-word sets and entity-oriented templates. It supports supervised inference and remains effective with insufficient training data, motivating a separate zero-shot strategy.

  • 3 Prompt-learning for Entity Typing: A Naive Pipeline: Prompt-learning transfers fine-grained entity typing into masked language modeling, using prompts to learn and aggregate entity-type information.The pipeline includes label-word construction, template design, and supervised training.
  • 3.1 Label Words Set V∗: Hierarchical entity types are represented with label-word sets containing the words in each type, such as {location, city} for LOCATION/CITY.Related nouns such as “metropolis” and “town” can expand a type’s label-word set when they are unlikely to map to other entity types.
  • 3.1 Label Words Set V∗: Type probabilities aggregate masked-token confidence scores across the label words associated with each entity type.The importance parameter for each label word may be learned or defined heuristically during training.
  • 3.2 Templates: The method uses hard-encoding templates with natural-language prompts and soft-encoding templates with trainable special tokens.Hard templates copy the marked entity mention and append linking verbs, articles, and a [MASK] token; soft templates add prompt tokens whose embeddings are optimized during training.
  • 3.2 Templates: Soft prompt tokens are randomly initialized and optimized as embeddings that can represent clusters of semantically similar vocabulary words.The soft template begins with a delimiter and a copy of the entity mention before adding the trainable tokens.
  • 3.3 Training and Inference: Both hard- and soft-encoding strategies jointly optimize the pretrained model and prompt parameters with cross-entropy loss.The trained model predicts the current input label from the words predicted at the [MASK] position using the type-probability formulation.
  • 3.3 Training and Inference: The supervised prompt-learning pipeline remains effective when training data are insufficient, including the few-shot setting.This result motivates examining the more extreme zero-shot scenario, where no training data are available.
  • 3.3 Training and Inference: In zero-shot typing, a randomly initialized classifier amounts to guessing, while direct prompt inference performs better than guessing but suffers a catastrophic decline.These limitations motivate the paper’s subsequent question of whether PLMs can predict entity types without explicit training data.

4 Self-supervised Prompt-learning for Zero-shot Entity Typing

Self-supervised prompt-learning uses unlabeled entity-linked data and a predefined label schema to align masked-token distributions for entity typing. The method samples positive and negative pairs, compares predictions over projected label words, and optimizes their similarity with a penalty for uncertain negative-pair assumptions.

  • 4.1 Overview: The self-supervised objective makes prediction distributions for similar examples as similar as possible over the projected vocabulary V∗.Restricting optimization to entity-type words excludes unrelated vocabulary items from the gradient.
  • 4.1 Overview: Figure 3 illustrates the pipeline for unlabeled data and a predefined label set, with V∗ representing label words projected from that set.The figure displays only the positive pair.
  • 4.1 Overview: The approach targets zero-shot entity typing by using unlabeled data, a predefined label schema, and prompt-learning to infer entity types.The model constructs label words from the schema and wraps each sentence with a hard-encoding template containing a masked token.
  • 4.2 Self-supervised Learning: Positive pairs share an entity mention, while negative pairs contain different marked mentions; the method compares distributions rather than forcing identical entity labels.This preserves context-sensitive types, since one entity may have different types across sentences.
  • 4.2 Self-supervised Learning: Negative examples are filtered with a dictionary so that selected sentence pairs contain entities of different known types.The dictionary-based restriction is intended to avoid false negative samples.
  • 4.2 Self-supervised Learning: Entity mentions are randomly hidden with probability α to reduce overfitting to entity names.The reported setting is α = 0.4, using the T3 template for the paired examples.
  • 4.2 Self-supervised Learning: Jensen-Shannon divergence measures similarity between the two masked-position prediction distributions over V∗.The distributions are denoted PV∗(w|x) and PV∗(w|x′), and the objective combines positive similarity with penalized negative-pair terms.
  • 4.2 Self-supervised Learning: The training data are generated from entity-linked English Wikipedia, with about 1 million positive and negative pairs each.The loss includes a penalty term γ because the negative-pair assumption is considered loose.

5 Experiments

Experiments evaluate prompt-learning for fine-grained entity typing across supervised, few-shot, and zero-shot settings on three benchmarks. Prompt-based methods generally improve over fine-tuning, with especially strong gains under limited supervision, though effects vary by dataset and prompt design.

  • Experimental setup: Experiments compare FT, PLET, and PLET (S) across fully supervised, few-shot, and zero-shot settings on three fine-grained entity typing datasets.The datasets are FEW-NERD, OntoNotes, and BBN; evaluation uses Acc, MaF, and MiF.
  • Fully supervised results: Prompt-based methods improve over directly fine-tuned models across the three datasets, helping capture entity-type information from context.The fully supervised comparison uses BERT-based FT and prompt-learning variants with hard or soft encoding.
  • Analysis and limitations: Prompt-learning gains and encoding preferences vary across datasets; soft encoding is strongest on OntoNotes, whereas hard encoding is favored on FEW-NERD and BBN.The authors attribute part of this variation to dataset characteristics and prompt design.
  • Few-shot results: Prompt-learning outperforms fine-tuning by a large margin in few-shot settings, especially with only 1–2 training instances per entity type.The advantage persists when the number of entity types is large, ranging from 46 to 86.
  • Zero-shot results: Zero-shot PLET beats random guessing, while unlabeled-data training with PLET (S) improves performance by a large margin.Fine-tuning is not evaluated because a randomly initialized classifier cannot produce reasonable zero-shot results.
  • Template analysis: Template choice substantially affects few-shot performance, with location wording helping hard prompts and fewer special tokens benefiting soft prompts.The comparison is conducted under the 8-shot setting on FEW-NERD.

6 Related Work

Related work frames prompt-learning as a way to align downstream tasks with pre-training through cloze-style objectives. Prior research spans hand-crafted, automatically searched, and continuous prompts, while entity typing applies prompting to extract entity attributes.

  • Case analysis: The paper’s analysis includes zero-shot prediction distributions for four FEW-NERD types, comparing PLET with PLET (S).The figure distinguishes correct predictions from wrong predictions with correct or incorrect coarse-grained types.
  • Prompt-learning: Prompt-learning expresses downstream tasks as cloze-style objectives that resemble PLM pre-training, narrowing the objective-form gap of conventional fine-tuning.It can use manually designed or automatically generated prompts and does not require extra neural layers.
  • Prompt-learning: Prior prompt-learning research includes knowledge probing, relation classification, sentiment classification, and natural language inference.These studies use hand-crafted prompts, while later work explores automatic prompt search and continuous prompt embeddings.
  • Entity typing: Fine-grained entity typing assigns predefined types to marked entity mentions and is used here to stimulate PLMs to capture entity attributes.The paper positions this task as a knowledge-extraction setting for developing prompt-learning strategies.

7 Conclusion

The paper presents PLET for fine-grained entity typing across supervised, few-shot, and zero-shot settings, including a self-supervised extension for unlabeled data. It concludes that prompts exploit PLM prior knowledge while distribution-level optimization learns type information without overfitting.

  • Conclusion: PLET applies prompt-learning to fine-grained entity typing in fully supervised, few-shot, and zero-shot scenarios.Its self-supervised variant learns and summarizes entity types from unlabeled corpora and a predefined label schema.
  • Conclusion: PLET uses prompts and distribution-level optimization to exploit PLM prior knowledge and learn predefined type information without overfitting.The authors identify improved prompt-learning from unlabeled data as a direction for future work.
Loading 2108.10604v1…