Source-linked AI summary
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Guanghui Qin, Jason Eisner
TL;DR
The paper asks how prompts for extracting knowledge from pretrained language models should be obtained. It learns continuous soft prompts and mixtures of prompts by gradient descent, finding large improvements over prior methods and near-equivalent performance from random initialization. The approach exposes more of the models’ implicit factual knowledge while retaining practical flexibility in prompt design.
Problem
Prior work showed that language models contain factual knowledge elicited by prompts, but the prompts were largely manually designed.
Method
The paper learns continuous soft prompts and their mixture weights, jointly optimizing them with a differentiable objective using gradient-based training.
Results
The method produces startlingly large and consistent improvements over prior methods across cloze language models and factual and commonsense tasks, including with random initialization.
Takeaways & Limitations
Random initialization is nearly as good as informed initialization, indicating that effective prompts can be learned without relying strongly on prior prompt wording.
Takeaways & Limitations
The method’s demonstrated setting is cloze prompts for BERT and BART models producing single-word answers, and deeply perturbed prompts require controlling perturbation size through early stopping or regularization.
Abstract
from arXiv · showhide
Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to "fill in the blank" in a sentential prompt. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent -- either fine-tuning prompts taken from previous work, or starting from random initialization. Our prompts consist of "soft words," i.e., continuous vectors that are not necessarily word type embeddings from the language model. Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated. Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization.
1 Introduction
The paper learns prompts for querying pretrained language models by optimizing continuous soft vectors and mixtures of prompts. On factual and commonsense relations, the learned prompts outperform previous methods, including with random initialization.
- Gradient descent tunes prompts to better elicit the desired type of knowledge from language models.Unlike human questioning, LM prompts can be optimized directly in continuous vector space.
- Soft prompts relax the requirement that prompt vectors correspond to embeddings of actual English words.This makes optimization more convenient and allows prompts to adjust misleading, ambiguous, or overly specific wording.
- Mixtures of prompts provide broader coverage when multiple lexical patterns can express the same relation.The paper motivates combining alternatives such as “performed” and “painted” for date-of-death queries.
- The method is evaluated on cloze language models for factual and commonsense relations from three datasets.
- The method dramatically outperforms previous work on held-out examples, even when initialized randomly.The authors conclude that language models contain more extractable knowledge than prior prompting methods revealed.
2 Related Work
Prior work established that language models encode factual, commonsense, and other nonlinguistic abilities, usually elicited with manually designed prompts. This paper instead learns the prompts themselves, complementing approaches that modify models or use other probing mechanisms.
- Language modeling can implicitly acquire factual knowledge, commonsense, reasoning, summarization, and arithmetic abilities from training corpora.
- Most earlier approaches manually created prompts to extract answers from trained language models, with LAMA used as a baseline here.
- Some related methods tune soft tokens for text generation or fill-in-the-blank prompts in continuous space.These studies appeared after the paper’s submission and used pretrained GPT-2, BART, or BERT models.
- Other approaches mine prompts and fine-tune the language model, whereas this method tunes the prompts themselves.
- The approach could potentially extend to sentence-probing prompts and few-shot learning prompts.
3 Method
The method represents relational queries with hard or soft prompts, combines multiple soft prompts through learned mixture weights, and jointly optimizes prompts and weights by differentiable training. It can also condition prompt selection on the queried entity.
- The task predicts y from x for a fixed binary relation using training pairs and evaluates predictions on held-out pairs.
- A prompt is a sentence or phrase containing two blanks, with x inserted into one blank and the LM predicting a single-word y for the other.
- Soft prompts replace prompt words with arbitrary vectors that can be continuously tuned without changing their number or positions.They may be initialized from a hard prompt, creating a 6d-dimensional search space for a six-token prompt.
- Deep perturbations tune prompt representations across all LM layers, yielding 6d · (L + 1) parameters for a six-token prompt.Small perturbations can be encouraged through early stopping or regularization to preserve familiar activation patterns.
- The ensemble predictive distribution weights each prompt’s LM prediction by learned p(t | r) values that form a distribution over prompts.
- Prompt weights can depend on x, allowing the model to favor prompts suited to properties of the queried entity.The paper gives plural nouns and plural verbs as an example of entity-dependent prompt selection.
- Prompts and mixture weights are jointly optimized to minimize log-loss through a continuous differentiable objective.The objective can be locally minimized by backpropagation with gradient descent or by EM.
4 Experiments
The experiments evaluate learned prompt mixtures across several relation datasets, language models, initializations, and training variants. Tuning soft prompts consistently improves results, with mixtures and mined initialization often performing best.
- Experimental setup: Prompts are initialized from manually authored, paraphrased, mined, or random vectors, enabling comparisons between informed and random initialization.Random vectors preserve the original prompt length and blank positions.
- Main results: Training tuned prompts consistently produce large improvements, beating prior methods even when informed prompt words are replaced by random initial vectors.This evaluates generalization on held-out relation examples rather than only training examples.
- Ablations: Ablations show that tuning word vectors provides the major benefit, while tuning mixture weights alone also helps; soft prompts retain some gains under disjoint training and test labels.The ablation uses BERT-large on T-REx original.
- Main results: The winning T-REx systems always use mixtures of soft prompts initialized from mined prompts and tune all layers except with RoBERTa.This pattern holds across the evaluated language models, datasets, and metrics.
- Ablations: Data-dependent mixture weights have little effect because training learns to discard x information by setting the temperature T high.The mixture therefore remains effectively relation-dependent in this experiment.
5 Conclusion
The paper concludes that learned soft prompts substantially improve information extraction from pretrained language models, despite no longer being natural language. It also points toward soft prompting for few-shot generative prediction.
- Learned soft prompts produce startlingly large and consistent improvements for cloze queries of BERT and BART.
- The approach extracts information from pretrained language models using prompts that are optimized rather than manually crafted.
- Soft prompting could improve few-shot prediction with pretrained generative language models without fine-tuning the entire model.
A Statistics of Relational Databases
Table 8 presents statistics for the paper’s various relational databases.
- Table 8 presents statistics for the various relational databases.
- The relational-database statistics are collected in a single table.
- The passage directs readers to Table 8 for database statistics.
B Statistics of the Initial Prompts
Table 7 reports statistics for the prompts used to initialize the SoftPrompt model.
- Table 7 reports statistics for prompts used to initialize the SoftPrompt model.
- The initial prompts are summarized through statistics in Table 7.
- The table concerns prompts used for SoftPrompt model initialization.
C Visualization of Soft Prompts
Figure 1 visualizes a mixture of soft prompts tuned only at layer 0. The tuned prompts are difficult to interpret: nearby words suggest music, but the templates lack meaningful phrases and do not clearly explain the intended instrument association.
- Figure 1 visualizes a mixture of soft prompts when only layer 0 is tuned.
- The tuned tokens’ closest words are largely associated with music.
- The soft templates do not form meaningful phrases or clearly explain priming an instrument response for a musician.
D Entropy of the Mixture Model
The mixture model uses entropy to characterize how its learned weight distribution spreads across prompts. In practice, tuning sometimes retains one prompt, but on average retains several variant prompts.
- Entropy is defined over the learned mixture weights p(t | r) for prompts associated with relation r.
- The effective number of retained prompts is summarized using statistics reported in Table 10.
- In some cases, tuning the mixture weights selects a single prompt, but on average it retains several variant prompts.Figure 1 illustrates this multi-prompt behavior.
E Challenging dataset with distinct y’s
The distinct-y experiment removes overlap in target values across train, development, and test sets while preserving an approximately 80-10-10 split. The method again improves results, supporting generalization to novel y values.
- Train, development, and test sets contain no common y values, with an approximately 80-10-10 split.The experiment uses T-REx as the relational database.
- The method again improves results when evaluated on distinct-y splits.The paper presents this as evidence of generalizability.
- Figure 1 visualizes LPAQA mining prompts for relation P1303 Instrument on T-REx extended using BERT-large-cased with layer-0 token embeddings tuned.Prompts are ordered by decreasing mixture weight, and prompts after the first 12 have negligible contribution.
- The distinct-y experiment uses BERT-base-cased and BERT-large-cased language models with LPAQA mining-based prompts as initial prompts.The experiment is conducted on the T-REx original dataset.