PaperlayerView exact revision on arXiv

Source-linked AI summary

Factual Probing Is [MASK]: Learning vs. Learning to Recall

Zexuan Zhong, Dan Friedman, Danqi Chen

arXiv:2104.05240v2cs.CL

TL;DR

Factual probing aims to estimate facts encoded in pre-trained language models, but optimized prompts may also learn regularities from disjoint training data. The paper introduces continuous prompt optimization and control experiments, finding stronger prediction alongside evidence that training-data patterns confound probing results.

  • Problem

    Prompt-search methods may exploit regularities in training data, making probing accuracy difficult to interpret as a lower bound on facts encoded by the language model.

  • Method

    The paper introduces OPTIPROMPT, which directly optimizes prompts in continuous embedding space, and uses control experiments to distinguish learning from learning to recall.

  • Results

    6.4%: OPTIPROMPT outperforms the best previous prompt method on LAMA, while all data-driven prompt methods exploit useful information from training-data regularities.

  • Takeaways & Limitations

    Factual probing results reflect both information elicited from the language model and information recoverable from the prompt-search training distribution.

  • Takeaways & Limitations

    Optimized factual probes can encode new information from training data, leaving the preferred method for measuring factual knowledge unresolved.

Abstract

from arXiv · show

Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes. Subsequent work has attempted to tighten the estimate by searching for better prompts, using a disjoint set of facts as training data. In this work, we make two complementary contributions to better understand these factual probing techniques. First, we propose OptiPrompt, a novel and efficient method which directly optimizes in continuous embedding space. We find this simple method is able to predict an additional 6.4% of facts in the LAMA benchmark. Second, we raise a more important question: Can we really interpret these probing results as a lower bound? Is it possible that these prompt-search methods learn from the training data too? We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution, and all the existing prompt methods, including ours, are able to exploit them for better fact prediction. We conduct a set of control experiments to disentangle "learning" from "learning to recall", providing a more detailed picture of what different prompts can reveal about pre-trained language models.

1 Introduction

Factual probing uses prompts to test what pre-trained language models can predict, but prompt optimization may also learn regularities from its training data. The paper introduces OPTIPROMPT and control experiments to separate eliciting encoded facts from learning to recall distributional patterns.

  • Factual probing evaluates whether a pre-trained language model predicts objects for subjects and relations on held-out subject-object pairs.
  • OPTIPROMPT directly optimizes real-valued input vectors in embedding space rather than searching only over discrete input tokens.Manual prompts can initialize the optimization process.
  • Training data contains regularities in Wikidata relations that naive classifiers and data-driven prompt methods can exploit for fact prediction.This challenges the assumption that non-overlapping training facts cannot predict test facts.
  • Randomly initialized models can yield non-trivial fact predictions when prompt-search algorithms exploit class statistics and higher-order lexical regularities.
  • Control experiments partition examples into easy and hard subsets to assess which predictions are less likely to come from training-data regularities.OPTIPROMPT outperforms prior methods in both subsets.

2 Background: Prompting for Facts

LAMA probes factual knowledge by asking masked language models to predict single-token objects from subject-relation prompts. Later methods select or optimize prompts using disjoint training facts, introducing increasingly learned prompt strategies.

  • 2.1 LAMA: LAMA represents facts as subject-relation-object triples drawn from sources including Wikidata, ConceptNet, and SQuAD.
  • 2.1 LAMA: The T-REx split contains up to 1000 triples for each of 41 Wikidata relation types, grouped into 1-1, N-1, and N-M categories.
  • 2.1 LAMA: LAMA associates each relation with a human-written single-[MASK] prompt and restricts evaluation to objects represented by one vocabulary token.
  • 2.1 LAMA: A language model is credited with encoding a fact when its highest-probability masked token matches the gold object.The prediction is computed as the object maximizing P([MASK] = ô | t_r(s)).
  • 2.1 LAMA: LAMA has no training data and evaluates pre-trained language models off the shelf, so manual prompts provide only a lower-bound estimate.
  • 2.2 LPAQA: LPAQA generates candidate prompts through text mining and paraphrasing, then selects prompts using accuracy on disjoint Wikidata training facts.Its methods include top-K baselines and an optimized ensemble with multiple weighted prompts.
  • 2.2 LPAQA: Subject names average 3.7 tokens in the BERT-base-cased vocabulary.
  • 2.3 AUTOPROMPT: AUTOPROMPT searches over vocabulary trigger tokens initialized as [MASK] tokens and iteratively updated with gradient-based likelihood estimates.The prompt places a fixed number of trigger tokens after the subject.

3 Our Approach: OPTIPROMPT

OPTIPROMPT replaces discrete prompt-token search with continuous embedding optimization, using fixed language-model parameters and gradient descent to find effective relation-specific prompts. It improves over prior methods on LAMA and LAMA-UHN, with manual initialization consistently helping.

  • OPTIPROMPT: OPTIPROMPT searches directly over continuous embedding vectors instead of restricting prompts to vocabulary tokens.This removes the discrete search constraint and permits each prompt vector to be any vector in the language model’s embedding space.
  • OPTIPROMPT: Each [V]i is a dense vector matching the language model’s input-embedding dimension, with the number of vectors set to a hyperparameter.The method can also derive the number and positions of vectors from manual prompts.
  • Optimization: OPTIPROMPT keeps the language model fixed and uses gradient descent to minimize training-set negative log-likelihood.The loss is defined over relation-specific subject–object pairs after substituting subject tokens into each prompt template.
  • Results: 6.2%–7.1% higher performance on LAMA-UHN and 5.4%–6.4% on LAMA compared with prior models.The improvement is consistent across relation categories except 1-1, where the manual prompt performs best.
  • Initialization: Manual prompt initialization consistently improves performance, providing a useful starting point for the non-convex optimization.The implementation uses manually written prompts to determine vector positions and initialize them with corresponding pretrained embeddings.

4 Can We Trust Optimized Prompts?

The paper tests whether optimized prompts reveal facts encoded by BERT or also learn regularities from prompt-search training data. Control experiments show that training-data patterns can predict LAMA facts and that prompt methods exploit them, complicating interpretation of probing accuracy.

  • 4 Can We Trust Optimized Prompts?: Optimized prompts may exploit regularities in the underlying fact distribution, so higher probing accuracy need not indicate that BERT encodes more facts.The paper asks whether Wikidata training patterns can predict unseen facts and whether optimized prompts exploit those patterns in practice.
  • 4.1 Facts can be predicted from training data: Class priors and subject-token correlations provide two training-data signals for predicting object labels.The majority-class model estimates P(o | r), while Naive Bayes estimates correlations between subject tokens and object labels.
  • 4.1 Facts can be predicted from training data: Naive Bayes performs better than the majority-class model across all categories, showing that subject-token correlations add predictive information.The majority-class model already performs well on relations dominated by one object label, including native language and continent.
  • 4.2 Prompts can exploit training data: Random Model and Random Embeddings controls test whether prompt optimization can exploit training-data regularities without relying fully on pretrained lexical information.The Random Model uses randomly initialized parameters, while Random Embeddings reinitializes input embeddings and the tied output head; fine-tuning reinitialized BERT provides a stronger control.
  • 4.2 Prompts can exploit training data: OPTIPROMPT finds successful prompts even with randomly initialized parameters, although most correct predictions come from eliciting the majority class label.AUTOPROMPT obtains 0% accuracy in the Random Model setting, whereas OPTIPROMPT remains successful.
  • 4.2 Prompts can exploit training data: Optimized prompts over-predict majority labels, with AUTOPROMPT and OPTIPROMPT showing the strongest tendency; AUTOPROMPT exceeds 95% on six relations.Filtering prompts containing proper nouns or gold object labels does not prevent this behavior; one position-held prompt elicits bishop for 100% of LAMA examples.
  • 4.2 Prompts can exploit training data: The easy subset contains facts predictable from training data, while the hard subset contains the remaining facts; this partition helps assess which predictions are less likely to come from training data.Easy facts are identified using reinitialized or fine-tuned BERT controls and the Naive Bayes model.
  • 4.2 Prompts can exploit training data: LPAQA also encodes training-distribution regularities, with occupation prompts reflecting common Wikidata occupations and template changes increasing majority-label predictions.The evidence indicates that optimized prompts can learn new facts to some extent while exploiting distributional patterns.

5 How to Interpret Probing Results?

Control experiments show that factual probing accuracy can reflect both information elicited from the language model and regularities learned from training data. Partitioning examples into easy and hard subsets provides a more detailed view of these behaviors.

  • The analysis partitions LAMA into easy facts predictable by training-data controls and hard facts that remain after removing them.The easy subset estimates facts predictable from training data; the hard subset contains the remaining examples.
  • OPTIPROMPT outperforms other prompts on both easy and hard subsets.
  • Qualitative examples show AUTOPROMPT and OPTIPROMPT exploiting subject-name tokens and other training-data regularities.In easy examples, predictions can rely on tokens in the subject name; in hard examples, prompts can over-fit common object labels.
  • OPTIPROMPT correctly predicts difficult city facts where other prompts predict majority or geographically related labels.
  • The control analysis cannot establish that hard facts are unlearnable from training data or identify which strategy produced an individual prediction.Possible strategies include class priors, lexical correlations, lexical information from the language model, and entity-specific information encoded in the model.
  • Examining specific predictions reveals patterns that summary LAMA accuracy does not expose and provides evidence about what information the language model encodes.

6 Discussion

The discussion weighs continuous and discrete prompt optimization while emphasizing that both can exploit training-data regularities. It also identifies white-box access and attribution of prediction sources as important boundaries.

  • Continuous vs. discrete prompts: Both continuous and discrete prompt optimization can find prompts that exploit training data, and discrete prompts are often opaque.
  • Continuous vs. discrete prompts: Continuous prompting is preferred because it is easier and more efficient to optimize and performs better on easy and hard subsets.
  • Continuous vs. discrete prompts: OPTIPROMPT requires white-box language-model access to compute gradients, so discrete prompts remain necessary for models exposed only through APIs.
  • Learning vs. learning to recall: Prompt optimization results remain difficult to interpret because a prediction may arise from training data or information encoded in the language model.

7 Related Work

The work builds on factual and linguistic probing research, prompt optimization, and studies of shortcut learning. These lines of work motivate controls for distinguishing information extracted from representations from patterns learned by probes.

  • The study extends LAMA cloze-style factual probing and subsequent data-driven prompt optimization methods.
  • Prompting has also been studied as a method for few-shot learning with language models.
  • Related linguistic-probing work measures whether probes extract information from representations or learn annotation patterns from probe training data.
  • The study relates to shortcut-learning research showing that neural models can exploit shallow heuristics instead of underlying structure.

8 Conclusion

The paper introduces OPTIPROMPT and uses controls to show that prompt optimization can exploit information in training-data distributions. Comparing prompt behaviors provides a more detailed view of what factual probes reveal about language models.

  • OPTIPROMPT is a continuous prompt-optimization method that outperforms prior work on factual probing.It improves over the best previous prompt method by 6.4% on LAMA.
  • Prompt-optimization training data contains useful regularities, allowing search algorithms to recover facts even from randomly initialized models.
  • Comparing prompt methods across controls clarifies their behavior and what they can reveal about pre-trained language models.

Ethical Considerations

The authors caution that facts recovered through probing should not automatically be treated as genuine facts, because prompts can exploit lexical correlations and imbalanced distributions.

  • Probing outputs should not be considered real facts because statistical models can learn lexical correlations between entity names and object labels.The authors characterize this behavior as stereotype-learning rather than factual recovery.
  • The problem is more pronounced when the training distribution comes from imbalanced sources such as Wikidata.
  • Per-relation accuracy is needed to inspect the specific predictions made by each prompting method.

A.2 Exploiting Training Data

The control experiments show that optimized prompts can exploit majority-label regularities and other training-data patterns, complicating claims that probing accuracy reflects memorized facts.

  • Optimized prompts tend to over-predict the majority class label, especially AUTOPROMPT and OPTIPROMPT.
  • Discrete prompts can elicit a specific label from a language model even when the mechanism producing the prediction is obscure.
  • A single determiner, “a,” can shift BERT toward the majority label in an LPAQA prompt.The change is from the manual position prompt to an otherwise nearly identical LPAQA prompt.
  • 50.9% versus 9.5%: one optimized prompt predicts the majority label more often than the manual prompt and nearly twice the LAMA distribution rate of 27.3%.
  • In the Random Model setting, OPTIPROMPT’s correct predictions are virtually all attributable to the majority class, with languages spoken as a notable exception.For languages spoken, OPTIPROMPT sometimes classifies subjects as speaking English or French.
  • Optimized prompts perform better on LAMA-easy than LAMA-hard, but high easy-set accuracy can occur without encoding the fact distribution.OPTIPROMPT outperforms AUTOPROMPT in both subsets, while manual prompts also perform well on some easy relations.

B Implementation Details

The implementation fits simple probabilistic baselines to Wikidata-derived training data and specifies separate optimization procedures for OPTIPROMPT and AUTOPROMPT comparisons.

  • OPTIPROMPT training uses Adam, a linear scheduler with warmup ratio 0.1, 10 epochs, learning rate 3e-3, and batch size 16.
  • AUTOPROMPT is run for 1000 iterations per model to search for a relation-specific prompt.
  • The probabilistic models are fit to the Wikidata training data collected by Shin et al. (2020).
  • The Class Prior model predicts each relation’s most frequent object label, while Naive Bayes predicts from subject tokens and the relation.
  • The probability estimates use add-one smoothing.

C Comparing Pre-trained Language Models

The model comparison controls the evaluated vocabulary and test set, then examines how model scale and architecture affect probing performance and its interpretation.

  • The comparison evaluates BERT, RoBERTa, and ALBERT using at most 1000 TRE-x training samples per relation and single-token objects.The LAMA test set is downsampled so every evaluated object is a single token for all models.
  • BERT-large performs better on LAMA than BERT-base, but is marginally worse on the Random Model baseline.This baseline does not indicate that BERT-large is simply more capable of finding training-data patterns.
  • RoBERTa performs worse than BERT on LAMA with these prompts, although the difference decreases in the fine-tuning setting.RoBERTa performs better in the Random Model setting with fine-tuning, suggesting training-data exploitation may contribute to the difference.
  • The authors identify correlations between probing results and downstream task performance as a promising direction for future work.They also propose using such correlations to guide model selection.
Loading 2104.05240v2…