PaperlayerView exact revision on arXiv

Source-linked AI summary

Prototypical Verbalizer for Prompt-based Few-shot Tuning

Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, Zhiyuan Liu

arXiv:2203.09770v1cs.CLcs.LG

TL;DR

Few-shot prompt-based tuning depends on verbalizers that map model outputs to labels, but manual choices require task knowledge and automatic choices are challenging with limited data. ProtoVerb learns class prototypes from training instances through contrastive learning. It outperforms existing automatic verbalizers, especially in extreme scarcity, and also boosts prompt-based tuning on untuned PLMs.

  • Problem

    Manual verbalizers depend on domain knowledge and human effort, while finding suitable label words automatically remains challenging, particularly with scarce training data.

    Source: §1 · pp. 1–2

  • Method

    ProtoVerb constructs automatic verbalizers by learning class prototype vectors from training instances with contrastive learning.

    Source: §1 · p. 2

  • Results

    ProtoVerb outperforms existing automatic verbalizers, especially with extremely scarce data, and ProtoVerb+ improves prompt-based tuning; on DBPedia, one instance per class yields an 11.26% absolute accuracy improvement on untuned PLMs.

    Source: §1 · p. 2, §5.6 · p. 7

  • Takeaways & Limitations

    ProtoVerb+ offers a cost-efficient way to use training data with fixed PLMs, without the input-length limit noted for GPT-3 in-context learning.

    Source: §5.6 · p. 7

Abstract

from arXiv · show

Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging.In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https://github.com/thunlp/OpenPrompt.

1 Introduction

Few-shot adaptation makes effective verbalizer construction difficult: manual choices require task knowledge, while automatic methods struggle with scarce data. ProtoVerb learns class prototypes from training instances using contrastive learning, outperforming automatic verbalizers and also improving prompt-based tuning with manual verbalizers, including on untuned PLMs.

1 Introduction

  • Prompt-based tuning recasts inputs as cloze questions and maps predicted words to task labels through a verbalizer.This approach is presented as a way to bridge pre-training and downstream tasks for few-shot learning.Source: §1 · p. 1
  • Manual verbalizers require task-specific knowledge and human effort, while searching or optimizing label words in large spaces is difficult with little data.Prompt-based verbalizers bridge model outputs and final predictions.Source: §1 · p. 2
  • ProtoVerb learns class prototype vectors from training instances to construct verbalizers automatically, aiming to capture class-level semantics implicit in those instances.The prototypes are intended to form concepts similar to human-designed labels.Source: §1 · p. 2
  • Its contrastive objective clusters same-class instances, separates different-class instances, and encourages each prototype to represent its class.ProtoVerb learns continuous vectors directly from training instances as a flexible plug-in verbalizer.Source: §1 · p. 2
  • ProtoVerb significantly outperforms search-based and soft verbalizers, especially with extremely scarce training data, and consistently improves prompt-based tuning when added to manual verbalizers.The improvements also extend to untuned PLMs.Source: §1 · p. 2

2 Related Work

Prior work develops prompt-based tuning and several verbalizer designs, but automatic verbalizer construction remains difficult with few examples. ProtoVerb connects prototype-based few-shot learning to verbalizer construction.

2.1 Prompt-based Tuning

  • Prompt-based tuning has expanded across NLP tasks, but verbalizer design remains a less explored component of prompts.The section situates this work in prompt-based tuning research.Source: §2.1 · pp. 2–3

2.2 Verbalizer Design

  • Manual verbalizers rely on task knowledge, search-based verbalizers struggle to find suitable words with few examples, and soft verbalizers need data for optimization.ProtoVerb instead learns prototype vectors from scratch for few-shot tuning.Source: §2.2 · p. 3

2.3 Prototype-based Few-shot Learning

  • Prototype-based few-shot methods represent same-class instances with prototypes, motivating their use as semantic representatives in verbalizer construction.PCL (Li et al., 2021) inspires the proposed training objectives.Source: §2.3 · p. 3

3 Background

The background frames few-shot classification as predicting labels for examples from a limited support set. It contrasts conventional classifier-based fine-tuning with prompt-based tuning, which formulates prediction as a cloze task using verbalizer label words.

3 Background

  • In N-way K-shot classification, each of N classes has K support sentences, and the goal is to predict labels for sentences from the task label set.The setup is stated for adapting a pretrained language model to a downstream task.Source: §3 · p. 3

3.1 Fine-tuning

  • Conventional fine-tuning trains a classifier over the [CLS] representation to predict labels and tunes the classifier and language model by maximizing the labeled-example log likelihood.The classifier operates on the sentence representation produced by the language model.Source: §3.1 · p. 3

3.2 Prompt-based Tuning

  • Prompt-based tuning wraps input text in a template with a [MASK] position and uses verbalizer label words to calculate probabilities over labels.The verbalizer aggregates scores for the label words associated with each class.Source: §3.2 · p. 3

4 Prototypical Verbalizer

ProtoVerb learns class prototypes from training instances and uses them as verbalizers, projecting [MASK] representations into an embedding space. Contrastive objectives shape instance and prototype similarities, and inference assigns labels by comparing queries with the prototypes.

4 Prototypical Verbalizer

  • ProtoVerb learns prototypes directly from training instances and uses them as verbalizers to map PLM outputs to labels.It projects [MASK] hidden states into an embedding space for prototype learning.Source: §4 · pp. 3–4, Figure 2 · p. 4

4.1 Instance Representation and Similarity Function

  • A linear encoder maps each [MASK] hidden state to an instance representation, and cosine similarity measures instance similarity.The initial representation is the last-layer hidden state of the [MASK] token.Source: §4.1 · p. 4

4.2 Loss Function

  • InfoNCE training combines instance-instance and instance-prototype objectives to separate classes and place each prototype at its class center.The first objective raises within-class similarity over between-class similarity; the second raises similarity to the matching class prototype.Source: §4.2 · p. 4

4.3 Inference

  • At inference, ProtoVerb scores a query against prototypes and predicts the class with the highest probability.When combined with other verbalizers, it standardizes their logits and averages the scores.Source: §4.3 · p. 4

5 Experiments

Experiments evaluate ProtoVerb on topic classification and entity typing under few-shot settings, comparing it with manual and automatic verbalizers. ProtoVerb outperforms other automatic verbalizers, especially with very little data, and its ensemble improves prompt-based tuning, including on untuned PLMs.

5.1 Datasets and Templates

  • The benchmarks are AG’s News, Yahoo, and DBPedia for topic classification, plus FewNERD for entity typing.The study uses multiple fixed manual templates to focus comparisons on verbalizers.Source: §5.1 · p. 5

5.2 Experimental Settings

  • Few-shot experiments sample 1, 2, 4, 8, or 16 instances per class and evaluate accuracy, testing ProtoVerb as either a single or extra verbalizer.The single-verbalizer setting assumes no manual verbalizer; the extra-verbalizer setting appends ProtoVerb without including it in tuning.Source: §5.2 · p. 5

5.3 Implementation Details

  • The experiments use RoBERTa-large for topic classification and BERT-base for entity typing, with task-specific tuning schedules.Prototype learning uses 128-dimensional vectors optimized with Adam.Source: §5.3 · p. 5

5 Experiments

  • The experiments are designed to compare verbalizers within a shared prompt-based pipeline, with the verbalizer as the differing component.The section reports few-shot experiments and their results.Source: §5.4 · p. 5, §5 · p. 5

5.4 Baselines

  • Baselines include human-defined ManualVerb, vocabulary-searching SearchVerb, and trainable-token SoftVerb.SearchVerb selects words maximizing training-data likelihood; SoftVerb learns token embeddings with model tuning.Source: §5.4 · p. 5

5.5 Single Verbalizer Results

  • ProtoVerb remarkably and consistently outperforms SearchVerb and SoftVerb, especially with only 1 or 2 instances per class.With sufficient training data, it reaches scores comparable to or exceeding ManualVerb.Source: §5.5 · p. 6, Table 2 · p. 6

5.6 Multiple Verbalizer Results

  • ProtoVerb+ improves prompt-based tuning in most cases, while SearchVerb+ and SoftVerb+ seldom improve on ManualVerb.On untuned PLMs, ProtoVerb+ also boosts results across tasks; on DBPedia with one instance per class, it yields an 11.26% absolute accuracy improvement.Source: Table 3 · p. 6, §5.6 · p. 7

6 Analysis

The AG’s News analyses examine ProtoVerb with fixed PLMs, its loss components, its robustness to noisy labels, and the semantics captured by its prototypes. Results also compare learned prototypes with manual verbalizers.

6.1 Fixed Model Experiments

  • With fixed PLMs, ProtoVerb exceeds ManualVerb at two shots per class and outperforms SoftVerb and SearchVerb in low-shot settings.The figure reports mean accuracy with 95% confidence intervals on AG’s News.Source: §6.1 · p. 7, Figure 3 · p. 7

6.2 Ablation Study

  • Optimizing prototype embeddings with ProtoVerb’s losses improves performance and stability over mean instance embeddings, while adding Lins is beneficial.The ablation also indicates that Lins helps ProtoVerb learn instance embeddings.Source: §6.2 · pp. 7–8, Table 4 · p. 7

6.3 Robustness on Noisy Samples

  • ProtoVerb is more robust to noisy samples than baseline automatic verbalizers, although even a limited number of mislabeled samples can substantially reduce performance.The comparison reports accuracy drops for one, two, or three incorrectly labeled samples.Source: Table 5 · p. 7, §6.3 · p. 8

6.4 Prototype Discretization

  • With one example per class, prototype-nearest words are often topic-related proper nouns; with more examples, they include abstract class concepts.The authors interpret this shift as prototypes summarizing shared, class-level rather than entity-level semantics, supporting performance on unseen data.Source: §6.4 · p. 8, Table 6 · p. 8

6.5 Is ProtoVerb Similar with ManualVerb?

  • ProtoVerb’s similarity to corresponding manual verbalizers exceeds the four-class average of 0.25 and increases with more shots.Business and Sports score higher than World and Tech; the authors suggest diverse subtopics make the latter harder to summarize.Source: §6.5 · pp. 8–9, Figure 4 · p. 8

7 Conclusion

The conclusion presents ProtoVerb as an automatic verbalizer learned from training instances and reports gains both alone and alongside manual verbalizers. It also identifies extensions to non-tuning methods and soft templates as future work.

  • ProtoVerb learns class prototypes from training instances using contrastive learning to construct automatic verbalizers.Source: §7 · p. 9
  • As a single verbalizer, ProtoVerb outperforms state-of-the-art automatic verbalizers, and combining it with manual verbalizers consistently improves prompt-based tuning with minor effort.Source: §7 · p. 9
  • Future work will extend ProtoVerb to non-tuning PLM algorithms and soft-template prompt tuning, and investigate combining label words with prototypes.Source: §7 · p. 9

Ethical Considerations

The ethical considerations passage situates the work in few-shot settings and notes that released code and hyperparameters are intended to reduce reproduction effort.

A Templates

The appendix lists templates for topic classification, ontology classification, question classification, and fine-grained entity typing.

  • For AG’s News topic classification, the templates include “This topic is about [MASK]”; its four categories are World, Sports, Business, and Tech.Source: A Templates · p. 11
  • For DBPedia, the templates use article title and abstract text to classify the subject’s ontology class, with 14 classes in total.Source: A Templates · p. 11
  • Yahoo question classification uses AG’s News templates with “news” replaced by “question,” while FewNERD uses three templates for its 66 entity types.Source: A Templates · p. 11

Related papers

Loading 2203.09770v1…