Source-linked AI summary
Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View
Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu, Le Sun
TL;DR
Prompt-based probing is widely used to evaluate pretrained language models, but its results can be inaccurate, inconsistent, and unreliable. The paper uses causal analysis to examine three biases and proposes backdoor adjustment; it concludes that these biases warrant rethinking evaluation criteria for PLMs.
Problem
Prompt-based probing is widely used to assess PLM abilities, yet studies report inaccurate, inconsistent, or unreliable results, while its underlying biases require further understanding.
Source: §1 · pp. 1–2Method
Results
The analysis identifies prompt preference, instance verbalization, and sample disparity as biases that can distort prompt-based probing evaluations.
Source: §1 · p. 2, §6.2 · p. 7Takeaways & Limitations
The authors conclude that probing biases should inform dataset and framework design and prompt a rethink of criteria for identifying better PLMs.
Source: §9 · p. 9The authors identify extending the framework to prompt-tuning-based probing and all PLM-based evaluations as future work.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Prompt-based probing has been widely used in evaluating the abilities of pretrained language models (PLMs). Unfortunately, recent studies have discovered such an evaluation may be inaccurate, inconsistent and unreliable. Furthermore, the lack of understanding its inner workings, combined with its wide applicability, has the potential to lead to unforeseen risks for evaluating and applying PLMs in real-world applications. To discover, understand and quantify the risks, this paper investigates the prompt-based probing from a causal view, highlights three critical biases which could induce biased results and conclusions, and proposes to conduct debiasing via causal intervention. This paper provides valuable insights for the design of unbiased datasets, better probing frameworks and more reliable evaluations of pretrained language models. Furthermore, our conclusions also echo that we need to rethink the criteria for identifying better pretrained language models. We openly released the source code and data at https://github.com/c-box/causalEval.
1 Introduction
Prompt-based probing is widely used to assess pretrained language models, but its results can be inaccurate, inconsistent, and unreliable. The paper examines how linguistic and data-distribution correlations create evaluation biases and proposes a causal framework to identify, understand, and reduce them.
1 Introduction
- Prompt-based probing results can be inaccurate, inconsistent, and unreliable, risking mistaken conclusions about pretrained language models.The introduction links these risks to implicit connections among pretrained models, probing data, and prompts.Source: §1 · pp. 1–2
- Linguistic correlations and task-distribution overlap create implicit connections that can confound prompt-based evaluation.The introduction contrasts this with conventional evaluation, where tested hypotheses are raised independently of train/test data generation.Source: §1 · p. 2
- The paper proposes causal intervention via backdoor adjustment to reduce bias and improve probing reliability under stated assumptions.Its causal analysis framework is intended to identify, interpret, and eliminate biases.Source: §1 · pp. 2–3
- Three biases are identified: prompt preference, instance verbalization, and sample disparity.The framework investigates these biases and quantifies their risks in prompt-based evaluation.Source: §1 · p. 2
- The authors offer guidance for designing unbiased datasets and better probing frameworks, and argue that PLM evaluation criteria should be reconsidered.They explicitly do not aim to establish a universally correct probing criterion.Source: §1 · pp. 2–3
2 Background and Experimental Setup
The background introduces causal inference as a way to model relationships, identify confounding, and estimate causal effects through intervention. The experiments study factual-knowledge probing with LAMA and four pretrained language models.
2.1 Causal Inference
- Causal inference models relationships with a structural causal model (SCM), identifies confounders and spurious correlations, and uses intervention to estimate causal effects.A backdoor adjustment uses a set Z that blocks backdoor paths and contains no descendant of X; P(Z = z) can be estimated from data or given and is independent of X.Source: §2.1 · p. 3
2.2 Experimental Setup
- The study examines factual-knowledge probing, in which a model is queried to assess knowledge of facts such as a person's birthplace.For example, “Michael Jordan was born in [MASK]” uses the subject mention, a relation prompt, and a placeholder for the target object.Source: §2.2 · p. 3
- The experiments use LAMA, a Wikidata-derived dataset retaining 32 probing relations after removing relations unsuitable for P@1.Source: §2.2 · pp. 3–4
- The evaluation compares BERT, RoBERTa, GPT-2, and BART across autoencoder, autoregressive, and denoising-autoencoder architectures.Source: §2.2 · p. 4
3 Structural Causal Model for Factual Knowledge Probing
The paper represents factual-knowledge probing as a structural causal model linking pretraining, prompt and instance construction, and performance estimation. It identifies three backdoor paths that prevent observed model-performance associations from representing the causal effect of model choice.
Structural Causal Model
- The structural causal model includes 11 variables spanning pretraining distributions and corpora, models, linguistic and task variables, probing data, verbalized instances, and performance.These include the model M, prompt P, predictor I, verbalized instances X, and performance E.Source: §3 · p. 4
Causal Paths
- Pretraining samples a corpus from its distribution and linguistic distribution before training the model, while prompt selection expresses a relation and is influenced by linguistic distribution.Source: §3 · p. 4
- Probing instances are sampled by relation from the probing-data distribution and then verbalized according to the linguistic distribution.Source: §3 · p. 4
- Performance estimation combines a model and prompt into a predictor and evaluates it on verbalized instances; three backdoor paths mean observed model-performance associations do not represent the causal effect of model choice.The target effect is P(E|do(M = m), R = r), represented by the path M → I → E.Source: §3 · p. 4
4 Prompt Preference Bias
Prompt preference can make probing results depend on how well a prompt fits a model's linguistic preferences, undermining consistent performance and model comparisons. The paper traces this bias to a backdoor path and proposes blocking it with backdoor adjustment under a prior prompt-distribution formulation.
4 Prompt Preference Bias
- Because predictors combine a PLM with a prompt, prompt fitness can affect measured performance alongside the model's task ability.Source: §4 · p. 4
4.1 Prompt Preference Leads to Inconsistent Performance
- The demonstration uses at least five semantically equivalent, faithful prompts per relation that vary in linguistic expression.Source: §4.1 · p. 5
- For languages spoken, Precision@1 ranges from 3.90% to 65.44% on BERT-large and from 0.22% to 71.94% on BART-large across prompts.Figure 3 presents performance variation across semantically equivalent prompts for four PLMs and four relations.Source: §4.1 · p. 5, Figure 3 · p. 5
- PLM ranks are unstable on 96.88% of relations, and four prompts for owned by produce three distinct best and three distinct worst models.Source: §4.1 · p. 5, Figure 4 · p. 5
- On more than 70% of relations, best–worst prompt choice varies Precision@1 by >10 points, exceeding most performance gaps between models.Table 1 reports prompt-selection P@1 divergence averaged over relations.Source: Table 1 · p. 5, §4.1 · p. 5
- Prompt-driven performance inconsistency leads to unstable comparisons between PLMs and undermines prompt-based evaluations.Source: §4.1 · p. 5
4.2 Cause of Prompt Preference Bias
- A backdoor path links model M and performance E through shared correlation with linguistic distribution L, confounding model effects with prompt effects.Source: §4.2 · p. 5
- The proposed intervention blocks this path through backdoor adjustment, which requires a prior formulation of the prompt distribution P(P).Source: §4.2 · p. 5
5 Instance Verbalization Bias
Instance verbalization bias arises because different PLMs may prefer different verbalizations of the same probing instance, making predictions unstable. The section links this bias to linguistic confounding and describes causal intervention as a way to eliminate it.
5 Instance Verbalization Bias
- Different PLMs may prefer different verbalizations of the same instance, so changing an entity mention can change the evaluation result.For BERT, replacing “the U.S.” with “America” changes the predicted answer from Washington to Chicago.Source: §5 · p. 6, §5.1 · p. 6
5.1 Instance Verbalization Brings Unstable Predictions
- Average verbalization stability is below 40% for all four tested PLMs, indicating that predictions often change when verbalizations vary.Stability is the percentage of relation instances whose predictions remain unchanged across verbalizations.Source: §5.1 · p. 6
5.2 Cause of Instance Verbalization Bias
- Linguistic distribution confounds the relationship between the PLM and evaluation result through the path M ← C ← L → X → E.The observed association mixes the PLM’s effect with spurious correlation; blocking the path requires a distribution formulation for instance verbalizations, P(X).Source: §5.2 · p. 6
6 Sample Disparity Bias
Pretraining-corpus and probing-data distributional correlations can bias prompt-based comparisons between PLMs. The section illustrates this with LAMA and explains the backdoor path that can produce sample disparity bias.
6 Sample Disparity Bias
- Sample disparity bias means that performance differences between PLMs may reflect differences in their pretraining corpora rather than differences in ability.In prompt-based probing, each PLM is bundled with a unique pretraining corpus, and its correlation with the probing-data distribution cannot be quantified.Source: §6 · p. 6
6.1 Sample Disparity Brings Biased Performance
- Increasing γ in further-pretraining data mixtures improves performance, narrows the GPT-2–BERT gap when both use the same data, and WebText-only further pretraining undermines BERT.The experiment varies the WIKI-LAMA share in fixed-size 99k-sentence pretraining datasets and evaluates the models on LAMA.Source: §6.1 · p. 7
6.2 Cause of Sample Disparity Bias
- A backdoor path between model and evaluation result arises when pretraining-corpus and probing-data distributions are related, though the path’s form depends on their causal relationship.One common case is probing data sampled from a source such as Wikipedia that is also represented in pretraining data.Source: §6.2 · p. 7
7 Bias Elimination via Causal Intervention
Backdoor adjustment estimates the effect of a PLM on evaluation results by accounting for prompts and verbalized probing data. In experiments, intervention raised overall rank consistency from 5.5% to 68.5%.
7 Bias Elimination via Causal Intervention
- Because linguistic and data-distribution variables are unobservable, the adjustment uses prompts and verbalized instances, Z = {P, X}, to block backdoor paths.The adjustment weights conditional evaluation outcomes by P(p, x), the joint distribution of prompts and verbalized data.Source: §7 · p. 7
- The causal effect averages outcomes over the natural distributions of valid prompts and verbalized probing data, regardless of other factors.The exact joint distribution is intractable, so the paper estimates it by sampling prompts and verbalizations; this experiment assumes a uniform distribution.Source: §7 · pp. 7–8
- Overall rank consistency rose from 5.5% with randomly sampled prompts and verbalizations to 68.5% with causal intervention.The study evaluated consistency across 1,000 samples, each containing 20 LAMA relations, for eight PLMs.Source: §7 · p. 8, Table 4 · p. 8
8 Related Work
Prior NLP evaluation-bias research examined metrics, dataset artifacts, and spurious data-label correlations. This paper instead uses causal analysis to identify and address prompt-based probing biases, while leaving prompt-tuning outside its model.
8 Related Work
- Prompt-tuning is excluded from the paper’s causal model because it adds parameters and unknown correlations, and is left for future work.The discussion notes that prompt-tuning studies optimize prompts on training data, potentially at the cost of interpretability.Source: §8 · p. 8
- Earlier studies investigated metric biases, dataset artifacts, and spurious correlations between data and labels in NLP evaluation.The paper characterizes most prior work as empirical, with intuitive explanations of underlying causes.Source: §8 · pp. 8–9
- The paper applies a causal framework to identify, understand, and theoretically eliminate biases in prompt-based probing.The authors state that this framework can be adapted to other evaluation settings in a principled manner.Source: §8 · p. 9
9 Conclusions and Discussions
The paper identifies three biases in prompt-based probing and presents a causal framework for their identification, interpretation, and elimination. It calls for evaluation criteria that distinguish model architecture from pretraining corpus and identifies broader application of the framework as future work.
9 Conclusions and Discussions
- The paper identifies prompt preference, instance verbalization, and sample disparity biases and proposes a causal framework for identifying, interpreting, and eliminating them.The framework is presented with a theoretical guarantee.Source: §9 · p. 9
- Because pretraining corpus and architecture are bundled in pretrained language models, evaluations need to clarify which is being assessed and account for corpus–test data correlations.The paper says this motivates rethinking criteria for identifying better pretrained language models under prompt-based evaluation.Source: §9 · p. 9
- Extending the causal analysis framework to prompt-tuning-based probing and all PLM-based evaluations is identified as future work.Source: §9 · p. 9
Ethics Consideration
The paper reports no particular ethical consideration.
Ethics Consideration
- The paper reports no particular ethical consideration.Source: Ethics Consideration · p. 9
A Datasets Construction Details
The dataset construction filters instances and relations, and uses manually reviewed prompts fixed before experiments. The appendix also reports further-pretraining procedures for BERT and GPT2.
A Datasets Construction Details
- They removed instances with multi-token objects or objects outside the four-PLM vocabulary intersection, and excluded N-M relations because Precision@1 is unsuitable there and knowledge bases may omit correct answers.Source: A Datasets Construction Details · p. 13
- Prompts placed the object placeholder at the end, were manually filtered for semantic appropriateness, and were fixed before experiments.Prompt design accounted for differences among the four PLMs' pretraining tasks.Source: A Datasets Construction Details · p. 13
B Further Pretraining Details
- Further pretraining used BERT masked-language modeling with a 15% mask probability and GPT2 autoregressive modeling for three epochs on eight 40G-A100 GPUs, with maximum sequence length 512.Batch sizes were 256, 96, 128, and 64 for BERT-base, BERT-large, GPT2-base, and GPT2-medium; Adam used learning rate 5e−5 and warmup ratio 0.06.Source: B Further Pretraining Details · p. 13