Source-linked AI summary
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, Maosong Sun
TL;DR
Prompt-tuning can be effective in low-data classification, but verbalizers may have limited coverage and biased predictions. KPT expands verbalizers with external knowledge, refines the resulting words with the PLM, and improves error rates while reducing few-shot variance; external KB errors remain a limitation.
Problem
Prompt-tuning verbalizers can lack label-word coverage and introduce bias or high variance, limiting their reliability in low-data classification.
Method
KPT expands each class’s verbalizer with external knowledge, applies refinement methods to the expanded words, and aggregates their scores into label predictions.
Results
KPT reduces average classification error rates by 16%, 18%, 10%, and 7% in 0-, 1-, 5-, and 10-shot experiments, respectively, while reducing few-shot prediction variance.
Takeaways & Limitations
KPT improves and stabilizes prompt-tuning performance across zero-shot and few-shot text classification settings.
Takeaways & Limitations
External knowledge bases may contain errors or malicious words, so users should account for these risks.
Abstract
from arXiv · showhide
Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
1 Introduction
Prompt-tuning reframes classification as masked language modeling, but handcrafted or optimized verbalizers can lack coverage and introduce bias. KPT expands label words with external knowledge and refines them, improving error rates and stability in zero- and few-shot settings.
- Prompt-tuning: Extra classifiers make conventional fine-tuning challenging in few-shot and zero-shot learning because they require adequate training instances.Prompt-based methods bridge pre-training and downstream objectives in low-data settings.
- Prompt-tuning: Prompt-tuning wraps inputs in natural-language templates and predicts a masked token, connecting classification to masked language modeling.A verbalizer maps predicted label words to class labels.
- Verbalizer limitations: Manual verbalizers restrict predictions to limited label words, so relevant alternatives such as “physics” and “maths” are ignored.This limited coverage can introduce bias and makes optimal verbalizers difficult to construct.
- KPT motivation: Optimization-based expansion generally finds only a few words or embeddings close to the class name, making cross-granularity expansion difficult.KPT addresses this limitation by incorporating external knowledge into verbalizers, with particular suitability for zero-shot learning.
- KPT approach: KPT constructs label-word sets from external knowledge, applies four refinement methods, and uses vanilla or weighted averaging to map word scores to label scores.The refinement methods are frequency, relevance, contextualized calibration, and learnable refinement.
- Results: 16%, 18%, 10%, and 7% average classification error-rate reductions occur in 0-, 1-, 5-, and 10-shot experiments, respectively.KPT also consistently reduces prediction variances in few-shot experiments.
2 Related Work
Related work covers prompt-tuning and verbalizer construction. Existing verbalizers are manual, searched, or optimized, whereas KPT uses external knowledge to obtain diverse and comprehensive label-word sets.
- Prompt-tuning: Prompt-tuning has been applied broadly and can produce strong low-data performance for both large- and small-scale language models.The cited applications include text classification and other downstream tasks.
- Verbalizer construction: Verbalizers strongly influence prompt-tuning performance, but human-written choices can be vocabulary-biased and lack sufficient coverage.This motivates methods for automatically searching or learning label words.
- KPT overview: Figure 1 depicts KPT’s construction, refinement, and utilization processes, ending with a knowledgeable verbalizer that maps label-word predictions to labels.The figure presents the overall pipeline rather than a separate baseline method.
- Verbalizer construction: Automatic verbalizer-search methods require adequate training and validation data, while their outputs are usually synonyms of the class name.KPT instead expands each class with diverse label words from external knowledge bases.
3 Knowledgeable Prompt-tuning
KPT uses external knowledge to broaden prompt verbalizers, then refines noisy or biased label words before aggregating their predictions into class scores. Its construction and refinement procedures target coverage, relevance, rarity, and prior-probability problems.
- 3.1 Overview: Prompt-tuning wraps an input sequence with a template, predicts vocabulary words at [MASK], and maps selected word probabilities to label probabilities.The verbalizer partitions label words into class-specific sets and applies a transformation function g.
- 3.1 Overview: KPT generates multiple class-related label words from knowledge bases, removes noise through four refinement methods, and aggregates expanded-word scores with average-based approaches.Its utilization uses vanilla or weighted averages over the expanded label-word set.
- 3.2 Verbalizer Construction: External knowledge is used because masked contexts can admit many suitable words, making broad coverage and low subjective bias desirable verbalizer properties.The construction targets topic words across granularities and sentiment words across aspects and granularities.
- 3.2 Verbalizer Construction: For topic classification, KPT retrieves thresholded neighborhood words from a relevance-annotated knowledge graph; for sentiment, it uses a previously summarized sentiment dictionary.Each class’s label-word set includes its class name and related words.
- 3.3 Verbalizer Refinement: KPT refines KB-derived words because the KB vocabulary is not tailored to the PLM and can therefore contain noise.Frequency refinement removes rare words using contextualized priors, while relevance refinement evaluates class-specific relevance.
- 3.3 Verbalizer Refinement: Relevance refinement favors words more relevant to their own class than to other classes, removing words with R(v) < 1; its score is described as a TF-IDF variant.The method compares a word’s belonging-class relevance with average relevance to other classes.
- 3.3 Verbalizer Refinement: Contextualized calibration addresses biased predictions caused by differing label-word prior probabilities, which are more diverse for KB-derived words.It uses contextualized priors to calibrate the predicted distribution and normalizes the final probability.
4 Experiments
Experiments evaluate KPT and baselines on five text-classification datasets under zero- and few-shot settings, showing lower error rates and more stable predictions for KPT.
- Setup: Experiments cover three topic-classification datasets and two sentiment-classification datasets using prompt-based methods and fine-tuning baselines.The datasets are AG’s News, DBPedia, Yahoo, IMDB, and Amazon.
- Zero-shot: KPT variants consistently outperform PT and PT+CC in zero-shot experiments, while refinement methods improve the knowledgeable verbalizer.KPT without refinement exposes severe label noise in automatically constructed label words.
- Few-shot: Prompt-based methods outperform fine-tuning by a dramatic margin in nearly all few-shot settings, with the gap enlarging as the number of shots decreases.Automatic verbalizers lag behind other verbalizers especially in low-shot settings because label-word selection becomes inaccurate with limited labeled data.
- Few-shot: KPT reduces error rates from the best baselines by 17.8%, 10.3%, and 7.4% in 1-, 5-, and 10-shot experiments, respectively.KPT and its variants also generally achieve smaller variances than baseline methods.
- Few-shot: With 20 shots, performance gaps narrow as training data becomes sufficient, but KPT and its variants still outperform baselines by a consistent margin.The authors conjecture that noisy label words can still serve as training objectives for prompt tuning.
5 Analysis
The analysis examines KPT’s diverse label-word predictions and related implementation behavior, including calibration, refinement, and out-of-vocabulary handling.
- 5.1 Diversity of Top Predicted Words: KPT produces diverse label words across granularities rather than relying mainly on original class names.The case study counts label words appearing in top-5 masked-token predictions for correctly classified examples.
- Additional analyses: The analysis also studies support-set size, the reduced need for calibration and frequency refinement in few-shot learning, OOV handling, and label-word retention during refinement.It additionally discusses applications when knowledge-base resources are unavailable.
6 Conclusion
The paper concludes that KPT expands prompt verbalizers with external knowledge and refines them, with experiments indicating potential in zero- and few-shot classification.
- Conclusion: KPT expands prompt-tuning verbalizers using an external knowledge base and introduces refinement methods for knowledgeable verbalizers.The paper identifies better KB–prompt integration and extension to text generation as future directions.
Contributions
The paper’s contributions span research conception, method design, experimentation, writing, revision, proofreading, and comments.
- Contributions: Zhiyuan Liu, Huadong Wang, and Shengding Hu proposed the idea and led the research, with Hu designing methods and conducting experiments.The author statement assigns writing, revision, proofreading, and commenting responsibilities across the team.
Ethical Considerations
KPT relies on external knowledge bases to construct its verbalizer, but those resources may contain errors or malicious words.
- External knowledge bases are used to construct the knowledgeable prompt-tuning verbalizer.
- Potential errors in external knowledge bases are a concern for KPT.
- External knowledge bases may inject malicious words into the verbalizer.
A Pilot Experiments
The paper analyzes prompt-tuning probabilities, refinement components, calibration, and the role of human knowledge in verbalizer construction. It also reports how support-set size and refinement affect practical use.
- Pilot Experiments: Manual class names provide useful human knowledge, motivating refinement of manual verbalizers rather than learning them entirely from scratch.Pilot experiments compare soft verbalizers initialized with class names against variants without manual verbalizers.
- Theoretical Illustration of KPT: The theoretical framework decomposes each label probability into relevance, label-word prior, and PLM prediction components.Relevance is estimated through Relevance or Learnable Refinement, while Contextualized Calibration estimates the label-word prior.
- Theoretical Illustration of KPT: KPT combines multiple label words per class with a shared backbone hypothesis, forming a verbalizer that integrates three components.
- Practical Issues of Refinement: Frequency Refinement filters label words in the lower half of contextualized prior probability rankings.
- Practical Issues of Refinement: About 50 support examples are sufficient for satisfying calibration, with contextualized calibration more effective for many classes and context-free calibration for few classes.
D.2 Supervised Data Ease the Need for Calibration.
The supervised few-shot setting is assumed to reduce the need for calibration and frequency refinement, while refinement still removes many label words and preserves broad coverage. External knowledge bases are not always available.
- Calibration and Frequency Refinement are omitted in few-shot learning because supervised data are assumed to train label-word probabilities adequately.
- Adding Contextualized Calibration and Frequency Refinement does not consistently improve few-shot performance, except in Yahoo.
- Refinement removes many expanded label words while retaining informative ones, with at least 100 remaining in the reported cases.
- When no suitable knowledge base exists, unlabeled corpora can be mined for potential label words using LOTClass, although combining it with KPT is beyond this work.
E Datasets and Templates
Experiments use five text-classification datasets and four manual templates, with dataset-specific wording and task formulations. The study also evaluates calibration and label-word restrictions while sampling Amazon’s test set for efficiency.
- Datasets: Five datasets cover topic classification—AG’s News, DBPedia, and Yahoo—and sentiment classification—IMDB and Amazon.
- Templates: Four manual templates are used because they incorporate expert knowledge, remain competitive with auto-generated templates, and apply in zero-shot settings.
- Evaluation: The experiments include contextualized calibration, frequency refinement, and restricting expanded label words to single PLM-vocabulary tokens.
- Templates: AG’s News templates modify PET patterns because capitalization is unsuitable for KB label words and some original templates are insufficiently informative.
- Datasets: DBPedia determines a subject’s topic or type from a paragraph paired with its title, despite potentially different paragraph emphasis.
- Evaluation: Amazon evaluation samples 10,000 of 400,000 test examples, with pilot experiments indicating tiny performance influence.
F Experimental Settings
Experimental settings mostly use default Huggingface Transformers parameters, with a specified learning rate for soft verbalizer embeddings.
- Hyper-parameters: Most hyper-parameters use the default settings from Huggingface Transformers.
- Hyper-parameters: The hyper-parameter configuration is listed in Table 8.
- Hyper-parameters: Soft verbalizer label-word embeddings use a learning rate of 3e−4 to encourage faster convergence.