Source-linked AI summary
How Can We Know What Language Models Know?
Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig
TL;DR
Manually chosen prompts can miss facts that language models contain, making single-prompt probing only a lower bound on their knowledge. The paper generates prompts through corpus mining and paraphrasing, combines them with ensemble methods, and improves LAMA accuracy from 31.1% to 39.6%.
Problem
Manually created prompts may fail to retrieve facts that language models know, so single-prompt results provide only a lower-bound estimate of contained knowledge.
Method
The paper mines and paraphrases diverse relation-specific prompts, then selects or ensembles them to query relational knowledge from language models.
Results
31.1% to 39.6%: combining diverse prompts improves accuracy on the LAMA benchmark.
Takeaways & Limitations
Language models are more knowledgeable than single-prompt results indicate, but their factual retrieval is sensitive to prompt wording.
Takeaways & Limitations
Mining-based generation can produce noisy prompts that are frequent but not indicative of the target relation.
Abstract
from arXiv · showhide
Recent work has presented intriguing results examining the knowledge contained in language models (LM) by having the LM fill in the blanks of prompts such as "Obama is a _ by profession". These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as "Obama worked as a _" may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA.
1 Introduction
Language models can answer factual and other understanding queries through natural-language prompts, but manually written prompts may under-retrieve their knowledge. The paper automatically generates and ensembles prompts, improving LAMA accuracy from 31.1% to 39.6%.
- Motivation: Language models are increasingly used for text understanding through natural-language queries that generate answers or select likely choices.Applications include factoid questions, commonsense queries, and relational factual knowledge extraction.
- Problem: Manually created prompts may be sub-optimal because training contexts can differ from the experimenter’s wording, causing known facts to be missed.Therefore, results from a single prompt provide only a lower-bound estimate of contained knowledge.
- Problem: The paper seeks a more accurate estimate of LM knowledge because tighter retrieval matters scientifically and can increase recall in knowledge extraction systems.
- Approach: The authors generate diverse prompts through mining and paraphrasing, then combine answers with lightweight ensembles.Mining derives prompts from corpus evidence, while paraphrasing expands seed prompts into semantically similar expressions.
- Results: 39.6%: ensembling diverse prompts further improves accuracy, while analysis suggests LMs are sensitive to how they are queried.The released LPAQA archive supports future experiments on probing LM knowledge.
2 Knowledge Retrieval from LMs
The paper treats LM knowledge retrieval as predicting an object in a relation triple from a cloze prompt. Because one manually designed prompt is not guaranteed to be optimal, it learns effective prompts from gold subject-object examples.
- Formulation: Unlike structured knowledge-base queries, LMs retrieve facts by assigning probabilities to words in natural-language prompts, without deterministic guarantees.
- Formulation: Factual knowledge is represented as triples ⟨x, r, y⟩, where x is the subject, r the relation, and y the object.
- Formulation: A relation-specific cloze prompt tr contains subject and object placeholders, and the LM predicts the missing object after the subject is inserted.
- Scope: The experiments predict single-token objects and focus on filling missing objects rather than reversing the direction to predict subjects.The directional choice follows the prior formulation for fair comparison.
- Evaluation: An LM is counted as knowing a fact when its prediction matches the ground-truth object y.A good prompt therefore elicits the correct object as often as possible.
- Motivation: Because prior work used one manually defined prompt, the paper learns effective prompts from gold subject-object pairs for each relation.
3 Prompt Generation
The paper generates relation-specific prompt candidates by mining corpus sentences and paraphrasing seed prompts. Mining broadens coverage but can introduce noise, whereas paraphrasing targets lexical diversity while preserving meaning.
- 3 Prompt Generation: Prompt generation constructs multiple relation-specific candidates so that at least some can elicit ground-truth objects effectively.
- 3.1 Mining-based Generation: Mining methods identify Wikipedia sentences containing related subjects and objects under a distant-supervision assumption.
- 3.1 Mining-based Generation: Middle-word prompts replace subjects and objects with placeholders while retaining the words between them, such as “x was born in y”.
- 3.1 Mining-based Generation: Dependency-based prompts use syntactic analysis to extract the phrase spanning the shortest dependency path between subject and object.This handles relations whose indicative words do not occur between the two entities.
- 3.1 Mining-based Generation: Mining requires no manually created prompts and can apply wherever subject-object pairs are available, but frequent extracted patterns may be noisy and weakly relational.An example of such noise is the pattern “x, y”.
- 3.2 Paraphrasing-based Generation: Paraphrasing expands a seed prompt into semantically similar expressions to improve lexical diversity while remaining relatively faithful to the original.For example, “x shares a border with y” can yield “x adjoins y”.
- 3.2 Paraphrasing-based Generation: Back-translation generates candidates by translating the seed into another language and translating them back, then ranks candidates by round-trip probability.The method retains the top T prompts from the B^2 candidates.
4 Prompt Selection and Ensembling
After generating candidates, the paper selects or combines prompts using training-set accuracy, rank-based averaging, or learned relation-specific weights. Larger ensembles trade concentration on accurate prompts for greater prompt diversity.
- 4 Prompt Selection and Ensembling: Generated prompts vary in effectiveness, so test-time use requires a selection or ensembling strategy.The paper presents three ways to use the candidate set.
- 4.1 Top-1 Prompt Selection: Top-1 selection chooses the prompt with the highest training-set accuracy for predicting ground-truth objects.Accuracy is computed over subject-object pairs for the relation.
- 4.2 Rank-based Ensemble: Multiple prompts can recover knowledge expressed in different training contexts because entity pairs may have appeared under different linguistic forms.
- 4.2 Rank-based Ensemble: Rank-based ensembling orders prompts by training accuracy and averages their log probabilities over the top K prompts.
- 4.2 Rank-based Ensemble: K controls the trade-off between concentrating on the most accurate prompts and increasing prompt diversity.Small K emphasizes accuracy; large K emphasizes diversity.
- 4.3 Optimized Ensemble: Equal weighting of the top K prompts can be sub-optimal because candidate reliability differs.
- 4.3 Optimized Ensemble: The optimized ensemble learns relation-specific prompt weights by parameterizing a softmax distribution and maximizing gold-object probability on training data.Each relation has its own T-dimensional parameter vector.
5 Main Experiments
Experiments show that automatically generated and ensembled prompts improve factual retrieval across models, metrics, datasets, and prompt-generation strategies, though gains vary by setting.
- Single Prompt Experiments: 31.1% to 34.1%: the best single generated prompt improves BERT-base micro-averaged accuracy over manual prompts.For BERT-large, accuracy rises from 32.3% to 39.4%.
- Mining Results: Mined prompts can substantially outperform manual prompts, including gains of 60.0% for religion and 22.7% for subclass_of.The largest gains occur when manual prompts use syntactically complicated or less common wording.
- Ensembling: 38.9% and 43.7%: optimized ensembles achieve these BERT-base and BERT-large micro-averaged accuracies, respectively.They outperform rank-based ensembles by a large margin.
- Ensembling: Ensembling multiple prompts almost always improves performance, but adding more prompts does not always help.Top-2 or top-3 usually works best for mined prompts, whereas top-5 is best for paraphrased prompts.
- Mining vs. Paraphrasing: Paraphrased prompts usually outperform mined prompts in rank-based ensembles, whereas mined prompts perform better in optimization-based ensembles.The authors conjecture that mined prompts vary more, making weighting especially important.
- Micro vs. Macro: 22.8% to 25.7%: optimization improves BERT-base macro-averaged accuracy, while macro accuracy remains substantially below micro accuracy.The reported text also states 25.7% to 30.1% on BERT-base, likely referring to another comparison, but preserves that wording as given.
- LAMA-UHN Evaluation: On LAMA-UHN, overall performance drops sharply, but optimized ensembles still outperform manual prompts by a large margin.This suggests the methods retrieve knowledge not inferable from surface forms.
- Google-RE: On Google-RE, diverse-prompt ensembles improve both BERT models, although gains are smaller because the subset has only three relations and one is particularly difficult.For the difficult birth-date relation, only one prompt yields non-zero accuracy.
6 Omitted Design Elements
The authors tested several additional prompt-generation and scoring methods but omitted them from the final design when they were ineffective or unstable. LM-aware refinement improved oracle performance, while ensemble performance dropped slightly, suggesting overfitting.
- Prompt optimization: Gradient-guided prompt optimization was unstable and often produced prompts in unnatural English.The authors therefore used hillclimbing, which repeatedly masks one token and replaces it with the most probable token conditioned on the others.
- Prompt optimization: Hillclimbing refined mined and manual prompts by modifying tokens from left to right until convergence.The method was applied on T-REx-train before evaluation on T-REx.
- LM-aware refinement: After refinement, oracle performance increased significantly, but both rank-based and optimization-based ensemble performance dropped slightly.This pattern indicates that some refined prompts may have over-fit the training set.
- Backward-probability scoring: A backward-probability feature was approximated using only the B most probable objects from forward probabilities during training and testing.The approximation addressed the large search space for objects.
- Backward-probability scoring: The backward-probability method produced only a small improvement, suggesting that diversity-promoting scoring may not be necessary for LM knowledge retrieval.The result was reported for BERT-base in Table 15.
7 Related Work
Prior work has analyzed neural representations through probing tasks and ablations, while factual-knowledge probing has used manually defined prompts. Other approaches add external symbolic knowledge or entity embeddings, whereas this work focuses on improving knowledge retrieval through prompts.
- Representation analyses use extrinsic probing tasks or model ablations to study linguistic properties and behavioral variation.
- Factual-knowledge probing follows work using manually defined prompts that may underestimate language models’ achievable performance.
- Related approaches integrate external knowledge bases or entity embeddings into language-generation systems and pretrained language models.
8 Conclusion
The paper examines how prompt choice affects factual-knowledge retrieval from language models and introduces systematic ways to generate diverse prompts. Combining these prompts improves retrieval accuracy and indicates that language models contain more knowledge than initial prompt-based estimates suggest.
- Mining-based and paraphrasing-based methods systematically generate diverse prompts for querying relational knowledge.
- 8%: Combining generated prompts improves factual knowledge retrieval accuracy over manually designed prompts.
- The analysis indicates that language models are more knowledgeable than initial prompt-based assessments suggest.